Inter prediction-based image coding method and device

ABSTRACT

An image decoding method according to the present document may comprise: a step for deriving an inter prediction mode from encoded information; a step for constructing reference pictures lists; a step for deriving motion information including reference picture indices for symmetric motion vector differences (SMVD) on the basis of reference pictures included in the reference picture lists; and a step for generating prediction samples on the basis of the motion information. The reference picture indices for SMVD may be derived on the basis of short-term reference pictures included in the reference picture lists.

BACKGROUND OF THE DISCLOSURE Field of the Disclosure

The present document is related to an inter prediction-based imagecoding method and device.

Related Art

Recently, demand for high-resolution, high-quality image/video such as4K or 8K or higher ultra high definition (UHD) image/video has increasedin various fields. As image/video data has high resolution and highquality, the amount of information or bits to be transmitted increasesrelative to the existing image/video data, and thus, transmitting imagedata using a medium such as an existing wired/wireless broadband line oran existing storage medium or storing image/video data using existingstorage medium increase transmission cost and storage cost.

In addition, interest and demand for immersive media such as virtualreality (VR) and artificial reality (AR) content or holograms hasrecently increased and broadcasting for image/video is havingcharacteristics different from reality images such as game images hasincreased.

Accordingly, a highly efficient image/video compression technology isrequired to effectively compress, transmit, store, and reproduceinformation of a high-resolution, high-quality image/video havingvarious characteristics as described above.

Also, inter prediction in image/video coding may include a procedure forsymmetric motion vector difference (SMVD) reference indices and/or aprocedure for a merge motion vector difference (MMVD). Consideringreference picture marking (for example, short-term or long-termreference), there is a discussion of applying a related technology to aprocedure for SMVD reference indices.

SUMMARY

According to one embodiment of the present disclosure, a method and adevice for improving image/video coding efficiency are provided.

According to one embodiment of the present disclosure, a method and adevice for performing inter prediction efficiently in an image/videocoding system are provided.

According to one embodiment of the present disclosure, a method and adevice for signaling information on a motion vector difference in interprediction are provided.

According to one embodiment of the present disclosure, a method and adevice for signaling information on L0 motion vector differences and L1motion vector differences are provided when bi-prediction is applied toa current block.

According to one embodiment of the present disclosure, a method and adevice for signaling an SMVD flag are provided.

According to one embodiment of the present disclosure, a method and anapparatus for deriving an L1 motion vector difference based on an SMVDflag are provided.

According to one embodiment of the present disclosure, a procedure forderiving SMVD reference indices may be performed based on referencepicture marking.

According to one embodiment of the present disclosure, a procedure forderiving SMVD reference indices may be performed using short-termreference pictures (marked as short-term references).

According to an embodiment of the present document, a video/imagedecoding method performed by a decoding apparatus is provided.

According to an embodiment of the present document, a decoding apparatusfor performing video/image decoding is provided.

According to an embodiment of the present document, a video/imageencoding method performed by an encoding apparatus is provided.

According to an embodiment of the present document, an encodingapparatus for performing video/image encoding is provided.

According to one embodiment of the present document, there is provided acomputer-readable digital storage medium in which encoded video/imageinformation, generated according to the video/image encoding methoddisclosed in at least one of the embodiments of the present document, isstored.

According to an embodiment of the present document, there is provided acomputer-readable digital storage medium in which encoded information orencoded video/image information, causing to perform the video/imagedecoding method disclosed in at least one of the embodiments of thepresent document by the decoding apparatus, is stored.

Advantageous Effects

According to the present disclosure, the overall image/video compressionefficiency may be improved.

According to the present disclosure, signaling of information on motionvector differences may be performed efficiently.

According to the present disclosure, L1 motion vector differences may beefficiently derived when bi-prediction is applied to a current block.

According to the present disclosure, complexity of a coding system maybe reduced by efficiently signaling the information used for deriving L1motion vector differences.

According to one embodiment of the present disclosure, efficient interprediction may be performed using short-term reference pictures forderiving a reference picture index of SMVD.

The technical effects achieved through specific embodiments of thepresent disclosure are not limited to those described above. Forexample, various other technical effects may be obtained, which may beunderstood or derived from the present disclosure by a person havingordinary skills in the related art. Therefore, specific effects of thepresent disclosure are not limited to the embodiments disclosedexplicitly in this document and may include various other effects thatmay be understood or derived from the technical characteristics of thepresent disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a video/image coding system to whichthe embodiments of the present document may be applied.

FIG. 2 is a diagram schematically illustrating a configuration of avideo/image encoding apparatus to which the embodiments of the presentdocument may be applied.

FIG. 3 is a schematic diagram illustrating a configuration of avideo/image decoding apparatus to which the embodiment(s) of the presentdisclosure may be applied.

FIG. 4 shows an example of a video/image encoding method based on interprediction.

FIG. 5 shows an example of a video/image decoding method based on interprediction.

FIG. 6 exemplarily shows an inter prediction procedure.

FIG. 7 illustrates a method for constructing an MVP candidate listaccording to the present disclosure.

FIG. 8 is a diagram for describing symmetric motion vector differences(SMVD).

FIG. 9 is a diagram for describing a method of deriving motion vectorsin inter prediction.

FIG. 10 illustrates an MVD derivation process of MMVD according to oneembodiment of the present disclosure.

FIG. 11 illustrates an MVD derivation process of MMVD according toanother embodiment of the present disclosure.

FIG. 12 illustrates an MVD derivation process of MMVD according to yetanother embodiment of the present disclosure.

FIG. 13 illustrates an MVD derivation process of MMVD according to oneembodiment of the present disclosure.

FIG. 14 illustrates an MVD derivation process of MMVD according to oneembodiment of the present disclosure.

FIGS. 15 and 16 illustrate a video/image encoding method and one exampleof a related component according to an embodiment(s) of the presentdisclosure.

FIGS. 17 and 18 illustrate an image/video decoding method and oneexample of a related component according to an embodiment(s) of thepresent disclosure.

FIG. 19 illustrates an example of a content streaming system to whichthe embodiments of the present disclosure may be applied.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

The present document may be modified in various forms, and specificembodiments thereof are described and shown in the drawings. However,the embodiments are not intended for limiting the present document. Theterms used in the following description are used to merely describespecific embodiments, but are not intended to limit the presentdocument. An expression of a singular number includes an expression ofthe plural number, so long as it is clearly read differently. The termssuch as “include” and “have” are intended to indicate that features,numbers, steps, operations, elements, components, or combinationsthereof used in the following description exist and it should be thusunderstood that the possibility of existence or addition of one or moredifferent features, numbers, steps, operations, elements, components, orcombinations thereof is not excluded.

Meanwhile, each configuration in the drawings described in the presentdocument is shown independently for the convenience of descriptionregarding different characteristic functions, and does not mean thateach configuration is implemented as separate hardware or separatesoftware. For example, two or more components among each component maybe combined to form one component, or one component may be divided intoa plurality of components. Embodiments in which each component isintegrated and/or separated are also included in the scope of thedisclosure of the present document.

Hereinafter, examples of the present embodiment are described in detailwith reference to the accompanying drawings. In addition, like referencenumerals are used to indicate like elements throughout the drawings, andthe same descriptions on the like elements are omitted.

FIG. 1 illustrates an example of a video/image coding system to whichthe embodiments of the present document may be applied.

Referring to FIG. 1 , a video/image coding system may include a firstdevice (a source device) and a second device (a reception device). Thesource device may transmit encoded video/image information or data tothe reception device through a digital storage medium or network in theform of a file or streaming.

The source device may include a video source, an encoding apparatus, anda transmitter. The receiving device may include a receiver, a decodingapparatus, and a renderer. The encoding apparatus may be called avideo/image encoding apparatus, and the decoding apparatus may be calleda video/image decoding apparatus. The transmitter may be included in theencoding apparatus. The receiver may be included in the decodingapparatus. The renderer may include a display, and the display may beconfigured as a separate device or an external component.

The video source may acquire video/image through a process of capturing,synthesizing, or generating the video/image. The video source mayinclude a video/image capture device and/or a video/image generatingdevice. The video/image capture device may include, for example, one ormore cameras, video/image archives including previously capturedvideo/images, and the like. The video/image generating device mayinclude, for example, computers, tablets and smartphones, and may(electronically) generate video/images. For example, a virtualvideo/image may be generated through a computer or the like. In thiscase, the video/image capturing process may be replaced by a process ofgenerating related data.

The encoding apparatus may encode input video/image. The encodingapparatus may perform a series of procedures such as prediction,transform, and quantization for compaction and coding efficiency. Theencoded data (encoded video/image information) may be output in the formof a bitstream.

The transmitter may transmit the encoded image/image information or dataoutput in the form of a bitstream to the receiver of the receivingdevice through a digital storage medium or a network in the form of afile or streaming. The digital storage medium may include variousstorage mediums such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, and thelike. The transmitter may include an element for generating a media filethrough a predetermined file format and may include an element fortransmission through a broadcast/communication network. The receiver mayreceive/extract the bitstream and transmit the received bitstream to thedecoding apparatus.

The decoding apparatus may decode the video/image by performing a seriesof procedures such as dequantization, inverse transform, and predictioncorresponding to the operation of the encoding apparatus.

The renderer may render the decoded video/image. The renderedvideo/image may be displayed through the display.

The present document relates to video/image coding. For example, amethod/embodiment disclosed in the present document may be applied to amethod disclosed in the versatile video coding (VVC) standard, theessential video coding (EVC) standard, the AOMedia Video 1 (AV1)standard, the 2nd generation of audio video coding standard (AVS2) orthe next generation video/image coding standard (e.g., H.267, H.268, orthe like).

The present document suggests various embodiments of video/image coding,and the above embodiments may also be performed in combination with eachother unless otherwise specified.

In the present document, a video may refer to a series of images overtime. A picture generally refers to the unit representing one image at aparticular time frame, and a slice/tile refers to the unit constitutinga part of the picture in terms of coding. A slice/tile may include oneor more coding tree units (CTUs). One picture may consist of one or moreslices/tiles. One picture may consist of one or more tile groups. Onetile group may include one or more tiles. A brick may represent arectangular region of CTU rows within a tile in a picture. A tile may bepartitioned into a multiple bricks, each of which may be constructedwith one or more CTU rows within the tile. A tile that is notpartitioned into multiple bricks may also be referred to as a brick. Abrick scan may represent a specific sequential ordering of CTUspartitioning a picture, wherein the CTUs may be ordered in a CTU rasterscan within a brick, and bricks within a tile may be orderedconsecutively in a raster scan of the bricks of the tile, and tiles in apicture may be ordered consecutively in a raster scan of the tiles ofthe picture. A tile is a rectangular region of CTUs within a particulartile column and a particular tile row in a picture. The tile column is arectangular region of CTUs having a height equal to the height of thepicture and a width specified by syntax elements in the pictureparameter set. The tile row is a rectangular region of CTUs having aheight specified by syntax elements in the picture parameter set and awidth equal to the width of the picture. A tile scan is a specificsequential ordering of CTUs partitioning a picture in which the CTUs areordered consecutively in CTU raster scan in a tile whereas tiles in apicture are ordered consecutively in a raster scan of the tiles of thepicture. A slice includes an integer number of bricks of a picture thatmay be exclusively contained in a single NAL unit. A slice may consistsof either a number of complete tiles or only a consecutive sequence ofcomplete bricks of one tile. In the present document, a tile group and aslice may be used in place of each other. For example, in the presentdocument, a tile group/tile group header may be referred to as aslice/slice header.

Meanwhile, one picture may be divided into two or more subpictures. Asubpicture may be a rectangular region of one or more slices within apicture.

A pixel or a pel may mean a smallest unit constituting one picture (orimage). Also, ‘sample’ may be used as a term corresponding to a pixel. Asample may generally represent a pixel or a value of a pixel, and mayrepresent only a pixel/pixel value of a luma component or only apixel/pixel value of a chroma component.

A unit may represent a basic unit of image processing. The unit mayinclude at least one of a specific region of the picture and informationrelated to the region. One unit may include one luma block and twochroma (ex. cb, cr) blocks. The unit may be used interchangeably withterms such as block or area in some cases. In a general case, an M×Nblock may include samples (or sample arrays) or a set (or array) oftransform coefficients of M columns and N rows. Alternatively, thesample may mean a pixel value in the spatial domain, and when such apixel value is transformed to the frequency domain, it may mean atransform coefficient in the frequency domain.

In the present document, “A or B” may mean “only A”, “only B” or “both Aand B”. In other words, “A or B” in the present document may beinterpreted as “A and/or B”. For example, in the present document “A, Bor C (A, B or C)” means “only A”, “only B”, “only C”, or “anycombination of A, B and C”.

A slash (/) or comma (comma) used in the present document may mean“and/or”. For example, “A/B” may mean “A and/or B”. Accordingly, “A/B”may mean “only A”, “only B”, or “both A and B”. For example, “A, B, C”may mean “A, B, or C”.

In the present document, “at least one of A and B” may mean “only A”,“only B” or “both A and B”. Also, in the present document, theexpression “at least one of A or B” or “at least one of A and/or B” maybe interpreted the same as “at least one of A and B”.

Also, in the present document, “at least one of A, B and C” means “onlyA”, “only B”, “only C”, or “any combination of A, B and C”. Also, “atleast one of A, B or C” or “at least one of A, B and/or C” may mean “atleast one of A, B and C”.

Also, parentheses used in the present document may mean “for example”.Specifically, when “prediction (intra prediction)” is indicated, “intraprediction” may be proposed as an example of “prediction”. In otherwords, “prediction” in the present document is not limited to “intraprediction”, and “intra prediction” may be proposed as an example of“prediction”. Also, even when “prediction (i.e., intra prediction)” isindicated, “intra prediction” may be proposed as an example of“prediction”.

Technical features that are individually described in one drawing in thepresent document may be implemented individually or simultaneously.

FIG. 2 is a diagram schematically illustrating a configuration of avideo/image encoding apparatus to which the embodiments of the presentdocument may be applied. Hereinafter, what is referred to as the videoencoding apparatus may include an image encoding apparatus.

Referring to FIG. 2 , the encoding apparatus 200 includes an imagepartitioner 210, a predictor 220, a residual processor 230, and anentropy encoder 240, an adder 250, a filter 260, and a memory 270. Thepredictor 220 may include an inter predictor 221 and an intra predictor222. The residual processor 230 may include a transformer 232, aquantizer 233, a dequantizer 234, and an inverse transformer 235. Theresidual processor 230 may further include a subtractor 231. The adder250 may be called a reconstructor or a reconstructed block generator.The image partitioner 210, the predictor 220, the residual processor230, the entropy encoder 240, the adder 250, and the filter 260 may beconfigured by at least one hardware component (ex. An encoder chipset orprocessor) according to an embodiment. In addition, the memory 270 mayinclude a decoded picture buffer (DPB) or may be configured by a digitalstorage medium. The hardware component may further include the memory270 as an internal/external component.

The image partitioner 210 may partition an input image (or a picture ora frame) input to the encoding apparatus 200 into one or moreprocessors. For example, the processor may be called a coding unit (CU).In this case, the coding unit may be recursively partitioned accordingto a quad-tree binary-tree ternary-tree (QTBTTT) structure from a codingtree unit (CTU) or a largest coding unit (LCU). For example, one codingunit may be partitioned into a plurality of coding units of a deeperdepth based on a quad tree structure, a binary tree structure, and/or aternary structure. In this case, for example, the quad tree structuremay be applied first and the binary tree structure and/or ternarystructure may be applied later. Alternatively, the binary tree structuremay be applied first. The coding procedure according to the presentdisclosure may be performed based on the final coding unit that is nolonger partitioned. In this case, the largest coding unit may be used asthe final coding unit based on coding efficiency according to imagecharacteristics, or if necessary, the coding unit may be recursivelypartitioned into coding units of deeper depth and a coding unit havingan optimal size may be used as the final coding unit. Here, the codingprocedure may include a procedure of prediction, transform, andreconstruction, which are described later. As another example, theprocessor may further include a prediction unit (PU) or a transform unit(TU). In this case, the prediction unit and the transform unit may besplit or partitioned from the aforementioned final coding unit. Theprediction unit may be a unit of sample prediction, and the transformunit may be a unit for deriving a transform coefficient and/or a unitfor deriving a residual signal from the transform coefficient.

The unit may be used interchangeably with terms such as block or area insome cases. In a general case, an M×N block may represent a set ofsamples or transform coefficients composed of M columns and N rows. Asample may generally represent a pixel or a value of a pixel, mayrepresent only a pixel/pixel value of a luma component or represent onlya pixel/pixel value of a chroma component. A sample may be used as aterm corresponding to one picture (or image) for a pixel or a pel.

In the encoding apparatus 200, a prediction signal (predicted block,prediction sample array) output from the inter predictor 221 or theintra predictor 222 is subtracted from an input image signal (originalblock, original sample array) to generate a residual signal residualblock, residual sample array), and the generated residual signal istransmitted to the transformer 232. In this case, as shown, a unit forsubtracting a prediction signal (predicted block, prediction samplearray) from the input image signal (original block, original samplearray) in the encoder 200 may be called a subtractor 231. The predictormay perform prediction on a block to be processed (hereinafter, referredto as a current block) and generate a predicted block includingprediction samples for the current block. The predictor may determinewhether intra prediction or inter prediction is applied on a currentblock or CU basis. As described later in the description of eachprediction mode, the predictor may generate various information relatedto prediction, such as prediction mode information, and transmit thegenerated information to the entropy encoder 240. The information on theprediction may be encoded in the entropy encoder 240 and output in theform of a bitstream.

The intra predictor 222 may predict the current block by referring tothe samples in the current picture. The referred samples may be locatedin the neighborhood of the current block or may be located apartaccording to the prediction mode. In the intra prediction, predictionmodes may include a plurality of non-directional modes and a pluralityof directional modes. The non-directional mode may include, for example,a DC mode and a planar mode. The directional mode may include, forexample, 33 directional prediction modes or 65 directional predictionmodes according to the degree of detail of the prediction direction.However, this is merely an example, more or less directional predictionmodes may be used depending on a setting. The intra predictor 222 maydetermine the prediction mode applied to the current block by using aprediction mode applied to a neighboring block.

The inter predictor 221 may derive a predicted block for the currentblock based on a reference block (reference sample array) specified by amotion vector on a reference picture. Here, in order to reduce theamount of motion information transmitted in the inter prediction mode,the motion information may be predicted in units of blocks, sub-blocks,or samples based on correlation of motion information between theneighboring block and the current block. The motion information mayinclude a motion vector and a reference picture index. The motioninformation may further include inter prediction direction (L0prediction, L1 prediction, Bi prediction, etc.) information. In the caseof inter prediction, the neighboring block may include a spatialneighboring block present in the current picture and a temporalneighboring block present in the reference picture. The referencepicture including the reference block and the reference pictureincluding the temporal neighboring block may be the same or different.The temporal neighboring block may be called a collocated referenceblock, a co-located CU (colCU), and the like, and the reference pictureincluding the temporal neighboring block may be called a collocatedpicture (colPic). For example, the inter predictor 221 may configure amotion information candidate list based on neighboring blocks andgenerate information indicating which candidate is used to derive amotion vector and/or a reference picture index of the current block.Inter prediction may be performed based on various prediction modes. Forexample, in the case of a skip mode and a merge mode, the interpredictor 221 may use motion information of the neighboring block asmotion information of the current block. In the skip mode, unlike themerge mode, the residual signal may not be transmitted. In the case ofthe motion vector prediction (MVP) mode, the motion vector of theneighboring block may be used as a motion vector predictor and themotion vector of the current block may be indicated by signaling amotion vector difference.

The predictor 220 may generate a prediction signal based on variousprediction methods described below. For example, the predictor may notonly apply intra prediction or inter prediction to predict one block butalso simultaneously apply both intra prediction and inter prediction.This may be called combined inter and intra prediction (CIIP). Inaddition, the predictor may be based on an intra block copy (IBC)prediction mode or a palette mode for prediction of a block. The IBCprediction mode or palette mode may be used for content image/videocoding of a game or the like, for example, screen content coding (SCC).The IBC basically performs prediction in the current picture but may beperformed similarly to inter prediction in that a reference block isderived in the current picture. That is, the IBC may use at least one ofthe inter prediction techniques described in the present disclosure. Thepalette mode may be considered as an example of intra coding or intraprediction. When the palette mode is applied, a sample value within apicture may be signaled based on information on the palette table andthe palette index.

The prediction signal generated by the predictor (including the interpredictor 221 and/or the intra predictor 222) may be used to generate areconstructed signal or to generate a residual signal. The transformer232 may generate transform coefficients by applying a transformtechnique to the residual signal. For example, the transform techniquemay include at least one of a discrete cosine transform (DCT), adiscrete sine transform (DST), a karhunen-loeve transform (KLT), agraph-based transform (GBT), or a conditionally non-linear transform(CNT). Here, the GBT means transform obtained from a graph whenrelationship information between pixels is represented by the graph. TheCNT refers to transform generated based on a prediction signal generatedusing all previously reconstructed pixels. In addition, the transformprocess may be applied to square pixel blocks having the same size ormay be applied to blocks having a variable size rather than square.

The quantizer 233 may quantize the transform coefficients and transmitthem to the entropy encoder 240 and the entropy encoder 240 may encodethe quantized signal (information on the quantized transformcoefficients) and output a bitstream. The information on the quantizedtransform coefficients may be referred to as residual information. Thequantizer 233 may rearrange block type quantized transform coefficientsinto a one-dimensional vector form based on a coefficient scanning orderand generate information on the quantized transform coefficients basedon the quantized transform coefficients in the one-dimensional vectorform. Information on transform coefficients may be generated. Theentropy encoder 240 may perform various encoding methods such as, forexample, exponential Golomb, context-adaptive variable length coding(CAVLC), context-adaptive binary arithmetic coding (CABAC), and thelike. The entropy encoder 240 may encode information necessary forvideo/image reconstruction other than quantized transform coefficients(ex. values of syntax elements, etc.) together or separately. Encodedinformation (ex. encoded video/image information) may be transmitted orstored in units of NALs (network abstraction layer) in the form of abitstream. The video/image information may further include informationon various parameter sets such as an adaptation parameter set (APS), apicture parameter set (PPS), a sequence parameter set (SPS), or a videoparameter set (VPS). In addition, the video/image information mayfurther include general constraint information. In the presentdisclosure, information and/or syntax elements transmitted/signaled fromthe encoding apparatus to the decoding apparatus may be included invideo/picture information. The video/image information may be encodedthrough the above-described encoding procedure and included in thebitstream. The bitstream may be transmitted over a network or may bestored in a digital storage medium. The network may include abroadcasting network and/or a communication network, and the digitalstorage medium may include various storage media such as USB, SD, CD,DVD, Blu-ray, HDD, SSD, and the like. A transmitter (not shown)transmitting a signal output from the entropy encoder 240 and/or astorage unit (not shown) storing the signal may be included asinternal/external element of the encoding apparatus 200, andalternatively, the transmitter may be included in the entropy encoder240.

The quantized transform coefficients output from the quantizer 233 maybe used to generate a prediction signal. For example, the residualsignal (residual block or residual samples) may be reconstructed byapplying dequantization and inverse transform to the quantized transformcoefficients through the dequantizer 234 and the inverse transformer235. The adder 250 adds the reconstructed residual signal to theprediction signal output from the inter predictor 221 or the intrapredictor 222 to generate a reconstructed signal (reconstructed picture,reconstructed block, reconstructed sample array). If there is noresidual for the block to be processed, such as a case where the skipmode is applied, the predicted block may be used as the reconstructedblock. The adder 250 may be called a reconstructor or a reconstructedblock generator. The generated reconstructed signal may be used forintra prediction of a next block to be processed in the current pictureand may be used for inter prediction of a next picture through filteringas described below.

Meanwhile, luma mapping with chroma scaling (LMCS) may be applied duringpicture encoding and/or reconstruction.

The filter 260 may improve subjective/objective image quality byapplying filtering to the reconstructed signal. For example, the filter260 may generate a modified reconstructed picture by applying variousfiltering methods to the reconstructed picture and store the modifiedreconstructed picture in the memory 270, specifically, a DPB of thememory 270. The various filtering methods may include, for example,deblocking filtering, a sample adaptive offset, an adaptive loop filter,a bilateral filter, and the like. The filter 260 may generate variousinformation related to the filtering and transmit the generatedinformation to the entropy encoder 240 as described later in thedescription of each filtering method. The information related to thefiltering may be encoded by the entropy encoder 240 and output in theform of a bitstream.

The modified reconstructed picture transmitted to the memory 270 may beused as the reference picture in the inter predictor 221. When the interprediction is applied through the encoding apparatus, predictionmismatch between the encoding apparatus 200 and the decoding apparatus300 may be avoided and encoding efficiency may be improved.

The DPB of the memory 270 DPB may store the modified reconstructedpicture for use as a reference picture in the inter predictor 221. Thememory 270 may store the motion information of the block from which themotion information in the current picture is derived (or encoded) and/orthe motion information of the blocks in the picture that have alreadybeen reconstructed. The stored motion information may be transmitted tothe inter predictor 221 and used as the motion information of thespatial neighboring block or the motion information of the temporalneighboring block. The memory 270 may store reconstructed samples ofreconstructed blocks in the current picture and may transfer thereconstructed samples to the intra predictor 222.

FIG. 3 is a schematic diagram illustrating a configuration of avideo/image decoding apparatus to which the embodiment(s) of the presentdisclosure may be applied.

Referring to FIG. 3 , the decoding apparatus 300 may include an entropydecoder 310, a residual processor 320, a predictor 330, an adder 340, afilter 350, a memory 360. The predictor 330 may include an interpredictor 331 and an intra predictor 332. The residual processor 320 mayinclude a dequantizer 321 and an inverse transformer 321. The entropydecoder 310, the residual processor 320, the predictor 330, the adder340, and the filter 350 may be configured by a hardware component (ex. Adecoder chipset or a processor) according to an embodiment. In addition,the memory 360 may include a decoded picture buffer (DPB) or may beconfigured by a digital storage medium. The hardware component mayfurther include the memory 360 as an internal/external component.

When a bitstream including video/image information is input, thedecoding apparatus 300 may reconstruct an image corresponding to aprocess in which the video/image information is processed in theencoding apparatus of FIG. 2 . For example, the decoding apparatus 300may derive units/blocks based on block partition related informationobtained from the bitstream. The decoding apparatus 300 may performdecoding using a processor applied in the encoding apparatus. Thus, theprocessor of decoding may be a coding unit, for example, and the codingunit may be partitioned according to a quad tree structure, binary treestructure and/or ternary tree structure from the coding tree unit or thelargest coding unit. One or more transform units may be derived from thecoding unit. The reconstructed image signal decoded and output throughthe decoding apparatus 300 may be reproduced through a reproducingapparatus.

The decoding apparatus 300 may receive a signal output from the encodingapparatus of FIG. 2 in the form of a bitstream, and the received signalmay be decoded through the entropy decoder 310. For example, the entropydecoder 310 may parse the bitstream to derive information (ex.video/image information) necessary for image reconstruction (or picturereconstruction). The video/image information may further includeinformation on various parameter sets such as an adaptation parameterset (APS), a picture parameter set (PPS), a sequence parameter set(SPS), or a video parameter set (VPS). In addition, the video/imageinformation may further include general constraint information. Thedecoding apparatus may further decode picture based on the informationon the parameter set and/or the general constraint information.Signaled/received information and/or syntax elements described later inthe present disclosure may be decoded may decode the decoding procedureand obtained from the bitstream. For example, the entropy decoder 310decodes the information in the bitstream based on a coding method suchas exponential Golomb coding, CAVLC, or CABAC, and output syntaxelements required for image reconstruction and quantized values oftransform coefficients for residual. More specifically, the CABACentropy decoding method may receive a bin corresponding to each syntaxelement in the bitstream, determine a context model using a decodingtarget syntax element information, decoding information of a decodingtarget block or information of a symbol/bin decoded in a previous stage,and perform an arithmetic decoding on the bin by predicting aprobability of occurrence of a bin according to the determined contextmodel, and generate a symbol corresponding to the value of each syntaxelement. In this case, the CABAC entropy decoding method may update thecontext model by using the information of the decoded symbol/bin for acontext model of a next symbol/bin after determining the context model.The information related to the prediction among the information decodedby the entropy decoder 310 may be provided to the predictor (the interpredictor 332 and the intra predictor 331), and the residual value onwhich the entropy decoding was performed in the entropy decoder 310,that is, the quantized transform coefficients and related parameterinformation, may be input to the residual processor 320. The residualprocessor 320 may derive the residual signal (the residual block, theresidual samples, the residual sample array). In addition, informationon filtering among information decoded by the entropy decoder 310 may beprovided to the filter 350. Meanwhile, a receiver (not shown) forreceiving a signal output from the encoding apparatus may be furtherconfigured as an internal/external element of the decoding apparatus300, or the receiver may be a component of the entropy decoder 310.Meanwhile, the decoding apparatus according to the present disclosuremay be referred to as a video/image/picture decoding apparatus, and thedecoding apparatus may be classified into an information decoder(video/image/picture information decoder) and a sample decoder(video/image/picture sample decoder). The information decoder mayinclude the entropy decoder 310, and the sample decoder may include atleast one of the dequantizer 321, the inverse transformer 322, the adder340, the filter 350, the memory 360, the inter predictor 332, and theintra predictor 331.

The dequantizer 321 may dequantize the quantized transform coefficientsand output the transform coefficients. The dequantizer 321 may rearrangethe quantized transform coefficients in the form of a two-dimensionalblock form. In this case, the rearrangement may be performed based onthe coefficient scanning order performed in the encoding apparatus. Thedequantizer 321 may perform dequantization on the quantized transformcoefficients by using a quantization parameter (ex. quantization stepsize information) and obtain transform coefficients.

The inverse transformer 322 inversely transforms the transformcoefficients to obtain a residual signal (residual block, residualsample array).

The predictor may perform prediction on the current block and generate apredicted block including prediction samples for the current block. Thepredictor may determine whether intra prediction or inter prediction isapplied to the current block based on the information on the predictionoutput from the entropy decoder 310 and may determine a specificintra/inter prediction mode.

The predictor 320 may generate a prediction signal based on variousprediction methods described below. For example, the predictor may notonly apply intra prediction or inter prediction to predict one block butalso simultaneously apply intra prediction and inter prediction. Thismay be called combined inter and intra prediction (CIIP). In addition,the predictor may be based on an intra block copy (IBC) prediction modeor a palette mode for prediction of a block. The IBC prediction mode orpalette mode may be used for content image/video coding of a game or thelike, for example, screen content coding (SCC). The IBC basicallyperforms prediction in the current picture but may be performedsimilarly to inter prediction in that a reference block is derived inthe current picture. That is, the IBC may use at least one of the interprediction techniques described in the present disclosure. The palettemode may be considered as an example of intra coding or intraprediction. When the palette mode is applied, a sample value within apicture may be signaled based on information on the palette table andthe palette index.

The intra predictor 331 may predict the current block by referring tothe samples in the current picture. The referred samples may be locatedin the neighborhood of the current block or may be located apartaccording to the prediction mode. In the intra prediction, predictionmodes may include a plurality of non-directional modes and a pluralityof directional modes. The intra predictor 331 may determine theprediction mode applied to the current block by using a prediction modeapplied to a neighboring block.

The inter predictor 332 may derive a predicted block for the currentblock based on a reference block (reference sample array) specified by amotion vector on a reference picture. In this case, in order to reducethe amount of motion information transmitted in the inter predictionmode, motion information may be predicted in units of blocks,sub-blocks, or samples based on correlation of motion informationbetween the neighboring block and the current block. The motioninformation may include a motion vector and a reference picture index.The motion information may further include inter prediction direction(L0 prediction, L1 prediction, Bi prediction, etc.) information. In thecase of inter prediction, the neighboring block may include a spatialneighboring block present in the current picture and a temporalneighboring block present in the reference picture. For example, theinter predictor 332 may configure a motion information candidate listbased on neighboring blocks and derive a motion vector of the currentblock and/or a reference picture index based on the received candidateselection information. Inter prediction may be performed based onvarious prediction modes, and the information on the prediction mayinclude information indicating a mode of inter prediction for thecurrent block.

The adder 340 may generate a reconstructed signal (reconstructedpicture, reconstructed block, reconstructed sample array) by adding theobtained residual signal to the prediction signal (predicted block,predicted sample array) output from the predictor (including the interpredictor 332 and/or the intra predictor 331). If there is no residualfor the block to be processed, such as when the skip mode is applied,the predicted block may be used as the reconstructed block.

The adder 340 may be called reconstructor or a reconstructed blockgenerator. The generated reconstructed signal may be used for intraprediction of a next block to be processed in the current picture, maybe output through filtering as described below, or may be used for interprediction of a next picture.

Meanwhile, luma mapping with chroma scaling (LMCS) may be applied in thepicture decoding process.

The filter 350 may improve subjective/objective image quality byapplying filtering to the reconstructed signal. For example, the filter350 may generate a modified reconstructed picture by applying variousfiltering methods to the reconstructed picture and store the modifiedreconstructed picture in the memory 360, specifically, a DPB of thememory 360. The various filtering methods may include, for example,deblocking filtering, a sample adaptive offset, an adaptive loop filter,a bilateral filter, and the like.

The (modified) reconstructed picture stored in the DPB of the memory 360may be used as a reference picture in the inter predictor 332. Thememory 360 may store the motion information of the block from which themotion information in the current picture is derived (or decoded) and/orthe motion information of the blocks in the picture that have alreadybeen reconstructed. The stored motion information may be transmitted tothe inter predictor 260 so as to be utilized as the motion informationof the spatial neighboring block or the motion information of thetemporal neighboring block. The memory 360 may store reconstructedsamples of reconstructed blocks in the current picture and transfer thereconstructed samples to the intra predictor 331.

In the present document, the embodiments described in the filter 260,the inter predictor 221, and the intra predictor 222 of the encodingapparatus 200 may be the same as or respectively applied to correspondto the filter 350, the inter predictor 332, and the intra predictor 331of the decoding apparatus 300. The same may also apply to the unit 332and the intra predictor 331.

As described above, in video coding, prediction is performed to increasecompression efficiency. Through this, it is possible to generate apredicted block including prediction samples for a current block, whichis a block to be coded. Here, the predicted block includes predictionsamples in a spatial domain (or pixel domain). The predicted block isderived equally from the encoding device and the decoding device, andthe encoding device decodes information (residual information) on theresidual between the original block and the predicted block, not theoriginal sample value of the original block itself. By signaling to thedevice, image coding efficiency can be increased. The decoding apparatusmay derive a residual block including residual samples based on theresidual information, and generate a reconstructed block includingreconstructed samples by summing the residual block and the predictedblock, and generate a reconstructed picture including reconstructedblocks.

The residual information may be generated through transformation andquantization processes. For example, the encoding apparatus may derive aresidual block between the original block and the predicted block, andperform a transform process on residual samples (residual sample array)included in the residual block to derive transform coefficients, andthen, by performing a quantization process on the transformcoefficients, derive quantized transform coefficients to signal theresidual related information to the decoding apparatus (via abitstream). Here, the residual information may include locationinformation, a transform technique, a transform kernel, and aquantization parameter, value information of the quantized transformcoefficients etc. The decoding apparatus may performdequantization/inverse transformation process based on the residualinformation and derive residual samples (or residual blocks). Thedecoding apparatus may generate a reconstructed picture based on thepredicted block and the residual block. The encoding apparatus may alsodequantize/inverse transform the quantized transform coefficients forreference for inter prediction of a later picture to derive a residualblock, and generate a reconstructed picture based thereon.

In the present document, at least one of quantization/dequantizationand/or transform/inverse transform may be omitted. When thequantization/dequantization is omitted, the quantized transformcoefficient may be referred to as a transform coefficient. When thetransform/inverse transform is omitted, the transform coefficients maybe called coefficients or residual coefficients, or may still be calledtransform coefficients for uniformity of expression.

In the present document, a quantized transform coefficient and atransform coefficient may be referred to as a transform coefficient anda scaled transform coefficient, respectively. In this case, the residualinformation may include information on transform coefficient(s), and theinformation on the transform coefficient(s) may be signaled throughresidual coding syntax. Transform coefficients may be derived based onthe residual information (or information on the transformcoefficient(s)), and scaled transform coefficients may be derivedthrough inverse transform (scaling) on the transform coefficients.Residual samples may be derived based on an inverse transform(transform) of the scaled transform coefficients. This may beapplied/expressed in other parts of the present document as well.

Intra prediction may refer to prediction that generates predictionsamples for the current block based on reference samples in a picture towhich the current block belongs (hereinafter, referred to as a currentpicture). When intra prediction is applied to the current block,neighboring reference samples to be used for intra prediction of thecurrent block may be derived. The neighboring reference samples of thecurrent block may include samples adjacent to the left boundary of thecurrent block having a size of nW×nH and a total of 2×nH samplesneighboring the bottom-left, samples adjacent to the top boundary of thecurrent block and a total of 2×nW samples neighboring the top-right, andone sample neighboring the top-left of the current block. Alternatively,the neighboring reference samples of the current block may include aplurality of upper neighboring samples and a plurality of leftneighboring samples. In addition, the neighboring reference samples ofthe current block may include a total of nH samples adjacent to theright boundary of the current block having a size of nW×nH, a total ofnW samples adjacent to the bottom boundary of the current block, and onesample neighboring (bottom-right) neighboring bottom-right of thecurrent block.

However, some of the neighboring reference samples of the current blockmay not be decoded yet or available. In this case, the decoder mayconfigure the neighboring reference samples to use for prediction bysubstituting the samples that are not available with the availablesamples. Alternatively, neighboring reference samples to be used forprediction may be configured through interpolation of the availablesamples.

When the neighboring reference samples are derived, (i) the predictionsample may be derived based on the average or interpolation ofneighboring reference samples of the current block, and (ii) theprediction sample may be derived based on the reference sample presentin a specific (prediction) direction for the prediction sample among theperiphery reference samples of the current block. The case of (i) may becalled non-directional mode or non-angular mode and the case of (ii) maybe called directional mode or angular mode.

Furthermore, the prediction sample may also be generated throughinterpolation between the second neighboring sample and the firstneighboring sample located in a direction opposite to the predictiondirection of the intra prediction mode of the current block based on theprediction sample of the current block among the neighboring referencesamples. The above case may be referred to as linear interpolation intraprediction (LIP). In addition, chroma prediction samples may begenerated based on luma samples using a linear model. This case may becalled LM mode.

In addition, a temporary prediction sample of the current block may bederived based on filtered neighboring reference samples, and at leastone reference sample derived according to the intra prediction modeamong the existing neighboring reference samples, that is, unfilteredneighboring reference samples, and the temporary prediction sample maybe weighted-summed to derive the prediction sample of the current block.The above case may be referred to as position dependent intra prediction(PDPC).

In addition, a reference sample line having the highest predictionaccuracy among the neighboring multi-reference sample lines of thecurrent block may be selected to derive the prediction sample by usingthe reference sample located in the prediction direction on thecorresponding line, and then the reference sample line used herein maybe indicated (signaled) to the decoding apparatus, thereby performingintra-prediction encoding. The above case may be referred to asmulti-reference line (MRL) intra prediction or MRL based intraprediction.

In addition, intra prediction may be performed based on the same intraprediction mode by dividing the current block into vertical orhorizontal subpartitions, and neighboring reference samples may bederived and used in the subpartition unit. That is, in this case, theintra prediction mode for the current block is equally applied to thesubpartitions, and the intra prediction performance may be improved insome cases by deriving and using the neighboring reference samples inthe subpartition unit. Such a prediction method may be called intrasub-partitions (ISP) or ISP based intra prediction.

The above-described intra prediction methods may be called an intraprediction type separately from the intra prediction mode. The intraprediction type may be called in various terms such as an intraprediction technique or an additional intra prediction mode. Forexample, the intra prediction type (or additional intra prediction mode)may include at least one of the above-described LIP, PDPC, MRL, and ISP.A general intra prediction method except for the specific intraprediction type such as LIP, PDPC, MRL, or ISP may be called a normalintra prediction type. The normal intra prediction type may be generallyapplied when the specific intra prediction type is not applied, andprediction may be performed based on the intra prediction mode describedabove. Meanwhile, post-filtering may be performed on the predictedsample derived as needed.

Specifically, the intra prediction procedure may include an intraprediction mode/type determination step, a neighboring reference samplederivation step, and an intra prediction mode/type based predictionsample derivation step. In addition, a post-filtering step may beperformed on the predicted sample derived as needed.

When intra prediction is applied, the intra prediction mode applied tothe current block may be determined using the intra prediction mode ofthe neighboring block. For example, the decoding apparatus may selectone of most probable mode (mpm) candidates of an mpm list derived basedon the intra prediction mode of the neighboring block (ex. left and/orupper neighboring blocks) of the current block based on the received mpmindex and select one of the other remaining intro prediction modes notincluded in the mpm candidates (and planar mode) based on the remainingintra prediction mode information. The mpm list may be configured toinclude or not include a planar mode as a candidate. For example, if thempm list includes the planar mode as a candidate, the mpm list may havesix candidates. If the mpm list does not include the planar mode as acandidate, the mpm list may have three candidates. When the mpm listdoes not include the planar mode as a candidate, a not planar flag (ex.intra_luma_not_planar_flag) indicating whether an intra prediction modeof the current block is not the planar mode may be signaled. Forexample, the mpm flag may be signaled first, and the mpm index and notplanar flag may be signaled when the value of the mpm flag is 1. Inaddition, the mpm index may be signaled when the value of the not planarflag is 1. Here, the mpm list is configured not to include the planarmode as a candidate does not is to signal the not planar flag first tocheck whether it is the planar mode first because the planar mode isalways considered as mpm.

For example, whether the intra prediction mode applied to the currentblock is in mpm candidates (and planar mode) or in remaining mode may beindicated based on the mpm flag (ex. Intra_luma_mpm_flag). A value 1 ofthe mpm flag may indicate that the intra prediction mode for the currentblock is within mpm candidates (and planar mode), and a value 0 of thempm flag may indicate that the intra prediction mode for the currentblock is not in the mpm candidates (and planar mode). The value 0 of thenot planar flag (ex.

Intra_luma_not_planar_flag) may indicate that the intra prediction modefor the current block is planar mode, and the value 1 of the not planarflag value may indicate that the intra prediction mode for the currentblock is not the planar mode. The mpm index may be signaled in the formof an mpm_idx or intra_luma_mpm_idx syntax element, and the remainingintra prediction mode information may be signaled in the form of arem_intra_luma_pred_mode or intra_luma_mpm_remainder syntax element. Forexample, the remaining intra prediction mode information may indexremaining intra prediction modes not included in the mpm candidates (andplanar mode) among all intra prediction modes in order of predictionmode number to indicate one of them. The intra prediction mode may be anintra prediction mode for a luma component (sample). Hereinafter, intraprediction mode information may include at least one of the mpm flag(ex. Intra_luma_mpm_flag), the not planar flag (ex.Intra_luma_not_planar_flag), the mpm index (ex. mpm_idx orintra_luma_mpm_idx), and the remaining intra prediction mode information(rem_intra_luma_pred_mode or intra_luma_mpm_remainder). In the presentdocument, the MPM list may be referred to in various terms such as MPMcandidate list and candModeList. When MIP is applied to the currentblock, a separate mpm flag (ex. intra_mip_mpm_flag), an mpm index (ex.intra_mip_mpm_idx), and remaining intra prediction mode information (ex.intra_mip_mpm_remainder) for MIP may be signaled and the not planar flagis not signaled.

In other words, in general, when block splitting is performed on animage, a current block and a neighboring block to be coded have similarimage characteristics. Therefore, the current block and the neighboringblock have a high probability of having the same or similar intraprediction mode. Thus, the encoder may use the intra prediction mode ofthe neighboring block to encode the intra prediction mode of the currentblock.

For example, the encoder/decoder may configure a list of most probablemodes (MPM) for the current block. The MPM list may also be referred toas an MPM candidate list. Herein, the MPM may refer to a mode used toimprove coding efficiency in consideration of similarity between thecurrent block and neighboring block in intra prediction mode coding. Asdescribed above, the MPM list may be configured to include the planarmode or may be configured to exclude the planar mode. For example, whenthe MPM list includes the planar mode, the number of candidates in theMPM list may be 6. And, if the MPM list does not include the planarmode, the number of candidates in the MPM list may be 5.

The encoder/decoder may configure an MPM list including 5 or 6 MPMs.

In order to configure the MPM list, three types of modes can beconsidered: default intra modes, neighbor intra modes, and the derivedintra modes.

For the neighboring intra modes, two neighboring blocks, i.e., a leftneighboring block and an upper neighboring block, may be considered.

As described above, if the MPM list is configured not to include theplanar mode, the planar mode is excluded from the list, and the numberof MPM list candidates may be set to 5.

In addition, the non-directional mode (or non-angular mode) among theintra prediction modes may include a DC mode based on the average ofneighboring reference samples of the current block or a planar modebased on interpolation.

When inter prediction is applied, the predictor of the encodingapparatus/decoding apparatus may derive a prediction sample byperforming inter prediction in units of blocks. Inter prediction may bea prediction derived in a manner that is dependent on data elements (ex.sample values or motion information) of picture(s) other than thecurrent picture. When inter prediction is applied to the current block,a predicted block (prediction sample array) for the current block may bederived based on a reference block (reference sample array) specified bya motion vector on the reference picture indicated by the referencepicture index. Here, in order to reduce the amount of motion informationtransmitted in the inter prediction mode, the motion information of thecurrent block may be predicted in units of blocks, subblocks, or samplesbased on correlation of motion information between the neighboring blockand the current block. The motion information may include a motionvector and a reference picture index. The motion information may furtherinclude inter prediction type (L0 prediction, L1 prediction, Biprediction, etc.) information. In the case of inter prediction, theneighboring block may include a spatial neighboring block present in thecurrent picture and a temporal neighboring block present in thereference picture. The reference picture including the reference blockand the reference picture including the temporal neighboring block maybe the same or different. The temporal neighboring block may be called acollocated reference block, a co-located CU (colCU), and the like, andthe reference picture including the temporal neighboring block may becalled a collocated picture (colPic). For example, a motion informationcandidate list may be configured based on neighboring blocks of thecurrent block, and flag or index information indicating which candidateis selected (used) may be signaled to derive a motion vector and/or areference picture index of the current block. Inter prediction may beperformed based on various prediction modes. For example, in the case ofa skip mode and a merge mode, the motion information of the currentblock may be the same as motion information of the neighboring block. Inthe skip mode, unlike the merge mode, the residual signal may not betransmitted. In the case of the motion vector prediction (MVP) mode, themotion vector of the selected neighboring block may be used as a motionvector predictor and the motion vector of the current block may besignaled. In this case, the motion vector of the current block may bederived using the sum of the motion vector predictor and the motionvector difference.

The motion information may include L0 motion information and/or L1motion information according to an inter prediction type (L0 prediction,L1 prediction, Bi prediction, etc.). The motion vector in the L0direction may be referred to as an L0 motion vector or MVL0, and themotion vector in the L1 direction may be referred to as an L1 motionvector or MVL1. Prediction based on the L0 motion vector may be calledL0 prediction, prediction based on the L1 motion vector may be called L1prediction, and prediction based on both the L0 motion vector and the L1motion vector may be called bi-prediction. Here, the L0 motion vectormay indicate a motion vector associated with the reference picture listL0 (L0), and the L1 motion vector may indicate a motion vectorassociated with the reference picture list L1 (L1). The referencepicture list L0 may include pictures that are earlier in output orderthan the current picture as reference pictures, and the referencepicture list L1 may include pictures that are later in the output orderthan the current picture. The previous pictures may be called forward(reference) pictures, and the subsequent pictures may be called reverse(reference) pictures. The reference picture list L0 may further includepictures that are later in the output order than the current picture asreference pictures. In this case, the previous pictures may be indexedfirst in the reference picture list L0 and the subsequent pictures maybe indexed later. The reference picture list L1 may further includeprevious pictures in the output order than the current picture asreference pictures. In this case, the subsequent pictures may be indexedfirst in the reference picture list 1 and the previous pictures may beindexed later. The output order may correspond to picture order count(POC) order.

A video/image encoding procedure based on inter prediction may include,for example, the following.

FIG. 4 shows an example of a video/image encoding method based on interprediction.

The encoding apparatus performs inter prediction on the current block(S400). The encoding apparatus may derive inter prediction mode andmotion information of the current block and generate prediction samplesof the current block. Here, the inter prediction mode determination, themotion information derivation, and the prediction samples generationprocedure may be performed simultaneously, or one procedure may beperformed before the other. For example, the inter predictor of theencoding apparatus may include a prediction mode determiner, a motioninformation deriver, and a prediction sample deriver. The predictionmode determiner may determine a prediction mode for the current block,the motion information deriver may derive motion information of thecurrent block, and prediction sample deriver may derive motion samplesof the current block. For example, the inter predictor of the encodingapparatus may search for a block similar to the current block in apredetermined region (search region) of reference pictures throughmotion estimation and derive a reference block whose difference to thecurrent block is a minimum or a predetermined reference or less. Basedon this, the inter predictor may derive a reference picture indexindicating a reference picture in which the reference block is locatedand derive a motion vector based on a position difference between thereference block and the current block. The encoding apparatus maydetermine a mode applied to the current block among various predictionmodes. The encoding apparatus may compare RD costs for the variousprediction modes and determine an optimal prediction mode for thecurrent block.

For example, when a skip mode or a merge mode is applied to the currentblock, the encoding apparatus may configure a merge candidate list to bedescribed later and derive a reference block whose difference to thecurrent block is minimum or a predetermined reference or less amongreference blocks indicated by the merge candidates included in the mergecandidate list. In this case, a merge candidate associated with thederived reference block may be selected, and merge index informationindicating the selected merge candidate may be generated and signaled tothe decoding apparatus. The motion information of the current block maybe derived using the motion information of the selected merge candidate.

As another example, when the (A)MVP mode is applied to the currentblock, the encoding apparatus may configure a (A)MVP candidate list tobe described later and use a motion vector of an mvp candidate selectedfrom among the mvp (motion vector predictor) candidates included in the(A)MVP candidate list, as mvp of the current block. In this case, forexample, a motion vector indicating the reference block derived by theabove-described motion estimation may be used as the motion vector ofthe current block, and an mvp candidate having a motion vector whosedifference to the motion vector of the current block, among the mvpcandidates, is smallest may be the selected mvp candidate. A motionvector difference (MVP) which is a difference from which the mvp wassubtracted may be derived from the motion vector of the current block.In this case, the information on the MVD may be signaled to the decodingapparatus. In addition, when the (A)MVP mode is applied, the value ofthe reference picture index may be configured as reference picture indexinformation and separately signaled to the decoding apparatus.

The encoding apparatus may derive residual samples based on theprediction samples (S410). The encoding apparatus may derive theresidual samples by comparing the original samples of the current blockwith the prediction samples.

The encoding apparatus encodes image information including predictioninformation and residual information (S420). The encoding apparatus mayoutput the encoded image information in the form of a bitstream. Theprediction information may include prediction mode information (ex. skipflag, merge flag or mode index) and information on motion information asinformation related to the prediction procedure. The information on themotion information may include candidate selection information (ex.merge index, mvp flag or mvp index) that is information for deriving amotion vector. In addition, the information on the motion informationmay include the information on the MVD and/or reference picture indexinformation described above. The information on the motion informationmay include information indicating whether L0 prediction, L1 prediction,or bi prediction is applied. The residual information is information onthe residual samples. The residual information may include informationon quantized transform coefficients for the residual samples.

The output bitstream may be stored in a (digital) storage medium anddelivered to the decoding apparatus, or may be delivered to the decodingapparatus via a network.

Meanwhile, as described above, the encoding apparatus may generate areconstructed picture (including the reconstructed samples and thereconstructed block) based on the reference samples and the residualsamples. This is to derive the same prediction result in the encodingapparatus as that performed in the decoding apparatus and because codingefficiency may be increased. Therefore, the encoding apparatus may storea reconstructed picture (or reconstructed samples, a reconstructedblock) in the memory and use it as a reference picture for interprediction. As described above, the in-loop filtering procedure may befurther applied to the reconstructed picture.

A video/image decoding procedure based on inter prediction may include,for example, the following.

FIG. 5 shows an example of a video/image decoding method based on interprediction.

Referring to FIG. 5 , the decoding apparatus may perform an operationcorresponding to the operation performed in the encoding apparatus. Thedecoding apparatus may perform prediction on the current block based onthe received prediction information and derive prediction samples.

Specifically, the decoding apparatus may determine a prediction mode forthe current block based on the received prediction information (S500).The decoding apparatus may determine which inter prediction mode isapplied to the current block based on the prediction mode information inthe prediction information.

For example, the decoding apparatus may determine whether the merge modeis applied to the current block or whether (A)MVP mode is determinedbased on the merge flag. Alternatively, one of various inter predictionmode candidates may be selected based on the mode index. The interprediction mode candidates may include a skip mode, a merge mode, and/or(A)MVP mode, or may include various inter prediction modes describedbelow.

The decoding apparatus derives motion information of the current blockbased on the determined inter prediction mode (S510). For example, whena skip mode or a merge mode is applied to the current block, thedecoding apparatus may configure a merge candidate list to be describedlater, and select one of the merge candidates included in the mergecandidate list. The selection may be performed based on the aboveselection information (merge index). The motion information of thecurrent block may be derived using the motion information of theselected merge candidate. The motion information of the selected mergecandidate may be used as motion information of the current block.

As another example, when the (A)MVP mode is applied to the currentblock, the decoding apparatus may configure an (A)MVP candidate list tobe described later and use a motion vector of an mvp candidate selectedfrom the mvp candidates included in the (A)MVP candidate list as mvp ofthe current block. The selection may be performed based on theabove-described selection information (mvp flag or mvp index). In thiscase, the MVD of the current block may be derived based on theinformation on the MVD, and the motion vector of the current block maybe derived based on mvp and the MVD of the current block. In addition, areference picture index of the current block may be derived based on thereference picture index information. A picture indicated by thereference picture index in the reference picture list for the currentblock may be derived as a reference picture referred for interprediction of the current block.

Meanwhile, as described below, motion information of the current blockmay be derived without configuring a candidate list, and in this case,motion information of the current block may be derived according to aprocedure disclosed in a prediction mode to be described later. In thiscase, the configuration of the candidate list as described above may beomitted.

The decoding apparatus may generate prediction samples for the currentblock based on the motion information of the current block (S520). Inthis case, the reference picture may be derived based on the referencepicture index of the current block, and the prediction samples of thecurrent block may be derived using the samples of the reference blockindicated by the motion vector of the current block on the referencepicture. In this case, as described below, a prediction sample filteringprocedure may be further performed on all or some of the predictionsamples of the current block.

For example, the inter predictor of the decoding apparatus may include aprediction mode determiner, a motion information deriver, and aprediction sample deriver. The prediction mode for the current block maybe determined based on the prediction mode information received from theprediction mode determiner, motion information (motion vector and/orreference picture index, etc.) of the current block may be derived basedon the information on the motion information received from the motioninformation deriver, and the prediction sample deriver may derive theprediction samples of the current block.

The decoding apparatus generates residual samples for the current blockbased on the received residual information (S530). The decodingapparatus may generate reconstructed samples for the current block basedon the prediction samples and the residual samples and generate areconstructed picture based on the reconstructed samples (S540).Thereafter, the in-loop filtering procedure or the like may be furtherapplied as described above.

FIG. 6 exemplarily shows an inter prediction procedure.

Referring to FIG. 6 , as described above, the inter prediction proceduremay include determining an inter prediction mode, deriving motioninformation according to the determined prediction mode, and performingprediction based on the derived motion information (generation of aprediction sample). The inter prediction procedure may be performed bythe encoding apparatus and the decoding apparatus as described above. Inthis document, a coding apparatus may include an encoding apparatusand/or a decoding apparatus.

Referring to FIG. 6 , the coding apparatus determines an interprediction mode for the current block (S600). Various inter predictionmodes may be used for prediction of the current block in the picture.For example, various modes, such as a merge mode, a skip mode, a motionvector prediction (MVP) mode, an affine mode, a subblock merge mode, anda merge with MVD (MMVD) mode, and the like may be used. A decoder sidemotion vector refinement (DMVR) mode, an adaptive motion vectorresolution (AMVR) mode, a bi-prediction with CU-level weight (BCW), abi-directional optical flow (BDOF), and the like may also be used asadditional modes additionally or instead. The affine mode may be calledan affine motion prediction mode. The MVP mode may be referred to asadvanced motion vector prediction (AMVP) mode. In this document, somemodes and/or motion information candidates derived by some modes may beincluded as one of motion information candidates of other modes. Forexample, an HMVP candidate may be added as a merge candidate in themerge/skip mode or may be added as an MVP candidate in the MVP mode.When the HMVP candidate is used as a motion information candidate in themerge mode or the skip mode, the HMVP candidate may be referred to as anHMVP merge candidate.

Prediction mode information indicating the inter prediction mode of thecurrent block may be signaled from the encoding apparatus to thedecoding apparatus. The prediction mode information may be included inthe bitstream and received by the decoding apparatus. The predictionmode information may include index information indicating one of aplurality of candidate modes. Alternatively, the inter prediction modemay be indicated through hierarchical signaling of flag information. Inthis case, the prediction mode information may include one or moreflags. For example, a skip flag may be signaled to indicate whether askip mode is applied, and if the skip mode is not applied, a merge flagmay be signaled to indicate whether a merge mode is applied, and if themerge mode is not applied, it is indicated to apply an MVP mode or aflag for additional classification may be further signaled. The affinemode may be signaled in an independent mode or may be signaled in a modedependent on the merge mode or the MVP mode. For example, the affinemode may include an affine merge mode and an affine MVP mode.

Meanwhile, information indicating whether the list0 (L0) prediction, thelist1 (L1) prediction, or the bi-prediction described above is used inthe current block (current coding unit) may be signaled in the currentblock. The information may be referred to as motion prediction directioninformation, inter prediction direction information or inter predictionindication information, and may be configured/encoded/signaled in theform of, for example, an inter_pred_idc syntax element. That is, theinter_pred_idc syntax element may indicate whether the aforementionedlist0 (L0) prediction, list1 (L1) prediction, or bi-prediction is usedfor the current block (current coding unit). In this document, for theconvenience of description, the inter prediction type (L0 prediction, L1prediction, or BI prediction) indicated by the inter_pred_idc syntaxelement may be indicated as a motion prediction direction. L0 predictionmay be represented as pred_L0, L1 prediction as pred_L1, andbi-prediction as pred_BI. For example, the following prediction type maybe indicated according to the value of the inter_pred_idc syntaxelement.

TABLE 1 Value of Name of inter_pred_idc inter_ ( cbWidth + ( cbWidth +pred_idc cbHeight ) != 8 cbHeight ) = = 8 0 PRED_L0 PRED_L0 1 PRED_L1PRED_L1 2 PRED_BI —

As described above, one picture may include one or more slices. Theslice may have one of slice types including intra (I) slice, predictive(P) slice, and bi-predictive (B) slice. The slice type may be indicatedbased on slice type information. For blocks in an I slice, interprediction may not be used for prediction and only intra prediction maybe used. Of course, even in this case, the original sample value may becoded and signaled without prediction. Intra prediction or interprediction may be used for blocks in a P slice, and only uni predictionmay be used when inter prediction is used. Meanwhile, intra predictionor inter prediction may be used for blocks in a B slice, and up to biprediction may be used when inter prediction is used.

L0 and L1 may include reference pictures that are previouslyencoded/decoded prior to the current picture. For example, L0 mayinclude reference pictures before and/or after the current picture inPOC order, and L1 may include reference pictures after and/or before thecurrent picture in POC order. In this case, L0 may be assigned a lowerreference picture index relative to previous reference pictures in thePOC order than the current reference pictures, and L1 may be assigned alower reference picture index relative to previous reference pictures inthe POC order than the current picture. In the case of B slice,bi-prediction may be applied, and in this case, unidirectionalbi-prediction may be applied or bidirectional bi-prediction may beapplied. The bidirectional bi-prediction may be called truebi-prediction.

The coding apparatus derives motion information for the current block(S610). The motion information derivation may be derived based on theinter prediction mode.

Inter prediction may be performed using motion information of thecurrent block. The encoding apparatus may derive optimal motioninformation for the current block through a motion estimation procedure.For example, the encoding apparatus may search for a similar referenceblock having a high correlation in units of fractional pixels within apredetermined search range in the reference picture using the originalblock in the original picture for the current block, thereby derivingmotion information. Similarity of blocks may be derived based on adifference of phase based sample values. For example, the similarity ofthe blocks may be calculated based on the SAD between the current block(or template of the current block) and the reference block (or templateof the reference block). In this case, motion information may be derivedbased on a reference block having the smallest SAD in the search area.The derived motion information may be signaled to the decoding apparatusaccording to various methods based on the inter prediction mode.

The coding apparatus performs inter prediction based on the motioninformation on the current block (S620). The coding apparatus may deriveprediction sample(s) for the current block based on the motioninformation. The current block including the prediction samples may bereferred to as a predicted block.

When the merge mode is applied, the motion information of the currentprediction block is not directly transmitted, and the motion informationof the current prediction block is derived using motion information of aneighboring prediction block. Therefore, the motion information of thecurrent prediction block may be indicated by transmitting flaginformation indicating that the merge mode is used and a merge indexindicating which neighboring prediction blocks are used. The merge modemay be called a regular merge mode.

The encoder must search a merge candidate block used to derive motioninformation of the current prediction block to perform the merge mode.For example, up to five merge candidate blocks may be used, but theembodiment(s) of the present document are not limited thereto. A maximumnumber of the merge candidate blocks may be transmitted in a sliceheader or a tile group header and the embodiment(s) of the presentdocument are not limited thereto. After finding the merge candidateblocks, the encoder may generate a merge candidate list and select amerge candidate block having the smallest cost among them as a finalmerge candidate block.

The merge candidate list may use, for example, five merge candidateblocks. For example, four spatial merge candidates and one temporalmerge candidate may be used. Hereinafter, the spatial merge candidate orthe spatial MVP candidate to be described later may be referred to asSMVP, and the temporal merge candidate or the temporal MVP candidate tobe described later may be referred to as TMVP.

Hereinafter, a method of constructing a merge candidate list isdescribed.

The coding apparatus (encoder/decoder) inserts spatial merge candidatesderived by searching for spatial neighboring blocks of the current blockinto the merge candidate list. For example, the spatial neighboringblocks may include a bottom left corner neighboring block, a leftneighboring block, a upper right corner neighboring block, an upperneighboring block, and an upper left corner neighboring block of thecurrent block. However, this is an example, and in addition to theabove-described spatial neighboring blocks, additional neighboringblocks such as a right neighboring block, a bottom neighboring block,and a bottom right neighboring block may be further used as the spatialneighboring blocks. The coding apparatus may detect available blocks bysearching the spatial neighboring blocks based on the priority, and mayderive motion information of the detected blocks as the spatial mergecandidates.

The coding apparatus inserts the temporal merge candidate derived bysearching the temporal neighboring block of the current block into themerge candidate list. The temporal neighboring block may be located on areference picture that is a picture different from the current picturein which the current block is located. The reference picture in whichthe temporal neighboring block is located may be called a collocatedpicture or a col picture. The temporal neighboring block may be searchedin order of the bottom right corner neighboring block and the bottomright center block of the co-located block for the current block on thecol picture. Meanwhile, when motion data compression is applied,specific motion information may be stored as representative motioninformation for each predetermined storage unit in the col picture. Inthis case, it is not necessary to store the motion information for allthe blocks in the predetermined storage unit, thereby obtaining a motiondata compression effect. In this case, the predetermined storage unitmay be previously determined, for example, in 16×16 sample units, 8×8sample units, or the like, or size information on the predeterminedstorage unit may be signaled from the encoder to the decoder. When themotion data compression is applied, motion information of the temporalneighboring block may be replaced with representative motion informationof the predetermined storage unit in which the temporal neighboringblock is located. That is, in this case, from an implementation point ofview, a predetermined value is arithmetically shifted to the right basedon coordinates (top left sample position) of the temporal neighboringblock, and thereafter, the temporal merge candidate may be derived basedon motion information of the prediction block covering an arithmeticallyleft shifted position. For example, in the case of a sample unit havingthe predetermined storage unit is 2ηx2n, if the coordinates of thetemporal neighboring block are (xTnb, yTnb), motion information of theprediction block located at the modified position ((xTnb>>n)<<n),(yTnb>>n)<<n)). Specifically, for example, in case where thepredetermined storage unit is a 16×16 sample unit, if the coordinates ofthe temporal neighboring block are (xTnb, yTnb), motion information ofthe prediction block located at modified position ((xTnb>>4)<<4),(yTnb>>4)<<4)) may be used for the temporal merge candidate. Or, forexample, in case where the predetermined storage unit is an 8×8 sampleunit, if the coordinates of the temporal neighboring block are (xTnb,yTnb), motion information of the prediction block located at themodified position ((xTnb>>3)<<3), (yTnb>>3)<<3)) may be used for thetemporal merge candidate.

The coding apparatus may determine whether the number of current mergecandidates is smaller than the maximum number of merge candidates. Themaximum number of merge candidates may be predefined or signaled fromthe encoder to the decoder. For example, the encoder may generateinformation on the maximum number of merge candidates, encode theinformation, and transmit the encoded information to the decoder in theform of a bitstream. If the maximum number of merge candidates is filledup, a subsequent candidate addition process may not be performed.

As a result of the checking, if the number of the current mergecandidates is smaller than the maximum number of merge candidates, thecoding apparatus inserts the additional merge candidate into the mergecandidate list.

As a result of the checking, if the number of the current mergecandidates is not smaller than the number of the maximum mergecandidates, the coding apparatus may terminate the construction of themerge candidate list. In this case, the encoder may select an optimalmerge candidate among merge candidates configuring the merge candidatelist based on a rate-distortion (RD) cost, and signal selectioninformation (ex. merge index) indicating the selected merge candidate tothe decoder. The decoder may select the optimal merge candidate based onthe merge candidate list and the selection information.

The motion information of the selected merge candidate may be used asthe motion information of the current block, and the prediction samplesof the current block may be derived based on the motion information ofthe current block. An encoder may derive residual samples of the currentblock based on the prediction samples, and may signal residualinformation on the residual samples to a decoder. The decoder maygenerate reconstructed samples based on the residual samples and thepredicted samples derived based on the residual information, andgenerate a reconstructed picture based thereon as described above.

When the skip mode is applied, the motion information of the currentblock may be derived in the same manner as that of the case where themerge mode is applied. However, when the skip mode is applied, theresidual signal for the corresponding block is omitted, and thusprediction samples may be used as reconstructed samples.

When the MVP mode is applied, a motion vector predictor (mvp) candidatelist may be generated using a motion vector of a reconstructed spatialneighboring block and/or a motion vector of a temporal neighboring block(or Col block). That is, the motion vector corresponding to thereconstructed spatial neighboring block and/or the motion vectorcorresponding to the temporal neighboring block may be used as a motionvector predictor candidate. When bi-prediction is applied, an mvpcandidate list for deriving L0 motion information and an mvp candidatelist for deriving L1 motion information may be generated and usedseparately. The above-described prediction information (or informationon the prediction) may include selection information (ex. MVP flag orMVP index) indicating an optimal motion vector predictor candidateselected from the motion vector predictor candidates included in thelist. In this case, the predictor may select a motion vector predictorof the current block from among the motion vector predictor candidatesincluded in the motion vector candidate list using the selectioninformation. The predictor of the encoding apparatus may obtain a motionvector difference (MVD) between the motion vector of the current blockand the motion vector predictor, encode the same, and output it in abitstream form. That is, the MVD may be obtained as a value obtained bysubtracting the motion vector predictor from the motion vector of thecurrent block. In this case, the predictor of the decoding apparatus mayobtain a motion vector difference included in the information on theprediction and derive the motion vector of the current block by addingthe motion vector difference and the motion vector predictor. Thepredictor of the decoding apparatus may obtain or derive a referencepicture index indicating the reference picture from the information onthe prediction.

FIG. 7 is a flow chart illustrating a method of constructing a motionvector predictor candidate list.

Referring to FIG. 7 , an embodiment may first search for a spatialcandidate block for motion vector prediction and insert it into theprediction candidate list (S700). Thereafter, an embodiment maydetermine whether the number of spatial candidate blocks is less thantwo (S710). For example, in an embodiment, when the number of spatialcandidate blocks is less than 2, a temporal candidate block may besearched for and additionally inserted into the prediction candidatelist (S720), and when the temporal candidate block is unavailable, azero motion vector may be used (S730). That is, the zero motion vectormay be additionally inserted into the prediction candidate list.Thereafter, an embodiment may end the construction of the preliminarycandidate list (S740). Alternatively, according to an embodiment, whenthe number of spatial candidate blocks is not less than two, theconstruction of the preliminary candidate list may be terminated. Here,the preliminary candidate list may indicate an MVP candidate list(S740).

Meanwhile, when the MVP mode is applied, the reference picture index maybe explicitly signaled. In this case, the reference picture indexrefidxL0 for the L0 prediction and the reference picture index refidxL1for the L1 prediction may be separately signaled. For example, when MVPmode is applied and BI prediction is applied, both information onrefidxL0 and information on refidxL1 may be signaled.

When the MVP mode is applied, as described above, the information on theMVD derived from the encoding apparatus may be signaled to the decodingapparatus. The information on the MVD may include, for example,information representing x and y components of the MVD absolute valueand the sign. In this case, information indicating whether the MVDabsolute value is greater than 0 and greater than 1, and the MVDremainder may be signaled step by step. For example, the informationindicating whether the MVD absolute value is greater than 1 may besignaled only when the value of the flag information indicating whetherthe MVD absolute value is greater than 0 is 1.

For example, the information on the MVD may be configured as thefollowing syntax, encoded in the encoding apparatus, and signaled to thedecoding apparatus.

TABLE 2   mvd_coding( x0, y0, refList ,cpIdx ) { Descriptor abs_mvd_greater0_flag[ 0 ] ae(v)  abs_mvd_greater0_flag[ 1 ] ae(v) if( abs_mvd_greater0_flag[ 0 ] )   abs_mvd_greater1_flag[ 0 ] ae(v) if( abs_mvd_greater0_flag[ 1 ] )   abs_mvd_greater1_flag[ 1 ] ae(v) if( abs_mvd_greater0_flag[ 0 ] ) {   if( abs_mvd_greater1_flag[ 0 ] )   abs_mvd_minus2[ 0 ] ae(v)   mvd_sign_flag[ 0 ] ae(v)  } if( abs_mvd_greater0_flag[ 1 ] ) {   if( abs_mvd_greater1_flag[ 1 ] )   abs_mvd_minus[ 1 ] ae(v)   mvd_sign_flag[ 1 ] ae(v)  } }

For example, in Table 2, the abs_mvd_greater0_flag syntax element mayindicate information on whether the difference MVD is greater than 0,and the abs_mvd_greater1_flag syntax element may indicate information onwhether the difference MVD is greater than 1. Also, the abs_mvd_minus2syntax element may indicate information about a value obtained by −2 tothe difference MVD, and the mvd_sign_flag syntax element may indicateinformation about the sign of the difference MVD. In addition, in Table2, [0] of each syntax element may indicate information on L0, and [1]may indicate information on L1.

For example, MVD[compIdx] may be derived based onabs_mvd_greater0_flag[compIdx] *(abs_mvd_minus2[compIdx]+2)(1−2*mvd_sign_flag[compIdx]). Here, compIdx (or cpIdx) represents anindex of each component and may have a value of 0 or 1. compIdx 0 mayindicte x component and compIdx 1 may indicate y component. However,this is merely an example and values of each component may be expressedby using a coordinate system other than the x and y coordinate systems.

Meanwhile, MVD (MVDL0) for L0 prediction and MVD (MVDL1) for L1prediction may be separately signaled, and the information on MVD mayinclude information on MVDL0 and/or information on MVDL1. For example,when the MVP mode is applied to the current block and BI prediction isapplied, both information on the MVDL0 and information on the MVDL1 maybe signaled.

FIG. 8 is a diagram for describing symmetric motion vector differences(SMVD).

When BI prediction is applied, symmetric MVD may be used inconsideration of coding efficiency. In this case, signaling of some ofthe motion information may be omitted. For example, when symmetric MVDis applied to the current block, information on refidxL0, information onrefidxL1, and information on MVDL1 may not be signaled from the encodingapparatus to the decoding apparatus and may be internally derived. Forexample, when MVP mode and BI prediction are applied to the currentblock, flag information (ex. symmetric MVD flag information orsym_mvd_flag syntax element) indicating whether to apply symmetric MVDmay be signaled, and when the value of the flag information is 1, thedecoding apparatus may determine that symmetric MVD is applied to thecurrent block.

When symmetric MVD mode is applied (i.e., the value of symmetric MVDflag information is 1), information on mvp_l0_flag, mvp_l1_flag, andMVDL0 may be explicitly signaled and, as described above, signaling ofinformation on refidxL0, information on refidxL1, and information onMVDL1 may be omitted and derived internally. For example, refidxL0 maybe derived as an index indicating a previous reference picture closestto the current picture in POC order in reference picture list 0 (whichmay be called list 0 or L0). refidxL1 may be derived as an indexindicating a next reference picture closest to the current picture inthe POC order in reference picture list 1 (which may be called list 1 orL1). Or, for example, refidxL0 and refidxL1 may both be derived as 0.Or, for example, the refidxL0 and refidxL1 may be derived as minimumindexes having the same POC difference in the relationship with thecurrent picture. Specifically, for example, when [POC of currentpicture]−[POC of first reference picture indicated by refidxL0] is afirst POC difference and [POC of the second reference picture indicatedby refidxL1] is a second POC difference, only if the first POCdifference and the second POC difference are the same, a value ofrefidxL0 indicating the first reference picture may be derived as avalue of refidxL0 of the current block and a value of refidxL1indicating the second reference picture may be derived as a value ofrefidxL1 of the current block. In addition, for example, when there area plurality of sets in which the first POC difference and the second POCdifference are the same, refidxL0 and refidxL1 of a set having theminimum difference may be derived as refidxL0 and refidxL1 of thecurrent block.

Referring to FIG. 8 , reference picture list 0, reference picture list1, and MVDL0 and MVDL1 are shown. Here, MVDL1 is symmetric with MVDL0.

MVDL1 may be derived as negative (−) MVDL0. For example, the final(improved or modified) motion information (motion vector; MV) for thecurrent block may be derived based on the following equation.

$\begin{matrix}\left\{ \begin{matrix}{\left( {{mvx_{0}},{mvy}_{0}} \right) = \left( {{{mvpx}_{0} + {mvdx}_{0}},{{mvpy}_{0} + {mvdy}_{0}}} \right)} \\{\left( {{mvx}_{1},{mvy}_{1}} \right)\  = \left( {{{mvpx}_{1} - {mvdx}_{0}},{{mvpy}_{1} - {mvdy}_{0}}} \right)}\end{matrix} \right. & \left\lbrack {{Eq}.1} \right\rbrack\end{matrix}$

In Equation 1, mvx₀ and mvy₀ may represent an x component and a ycomponent of L0 motion information or motion vector for L0 prediction,and mvx₁ and mvy₁ may represent an x component and a y component of L1motion information or motion vector for L1 prediction. Also, mvpx₀ andmvpy₀ may represent the x component and y component of the motion vectorpredictor for L0 prediction, and mvpx₁ and mvpy₁ may represent the xcomponent and y component of the motion vector predictor for L1prediction. Also, mvdx₀ and mvdy₀ may represent an x component and a ycomponent of a motion vector difference for L0 prediction.

Meanwhile, the MMVD mode is a method of applying motion vectordifference (MVD) to the merge mode, and motion information directly usedto generate prediction samples of the current block (i.e., the currentCU) may be implicitly derived. For example, an MMVD flag (i.e.,mmvd_flag) indicating whether to use MMVD for a current block (i.e., acurrent CU) may be signaled, and MMVD may be performed based on thisMMVD flag. When MMVD is applied to the current block (i.e., whenmmvd_flag is 1), additional information on MMVD may be signaled.

Here, the additional information on the MMVD includes a merge candidateflag (i.e., mmvd_cand_flag) indicating whether the first candidate orthe second candidate in the merge candidate list is used together withthe MVD, and a distance index (i.e., mmvd_distance_idx) for indicatingthe motion magnitude and a direction index (i.e., mmvd_direction_idx)for indicating a motion direction.

In the MMVD mode, two candidates located in the first and second entriesamong the candidates in the merge candidate list (i.e., the firstcandidate or the second candidate) may be used, and the two candidates(i.e., the first candidate or the second candidate) may be used. One ofthem may be used as the base MV. For example, a merge candidate flag(i.e., mmvd_cand_flag) may be signaled to indicate any one of twocandidates (i.e., the first candidate or the second candidate) in themerge candidate list.

Furthermore, distance index (i.e., mmvd_distance_idx) specifies motionmagnitude information and indicate the pre-defined offset from thestarting point. The offset may be added to either horizontal componentor vertical component of starting MV. The relation of distance index andpre-defined offset is specified in the following table.

TABLE 3 MmvdDistance[ x0 ][ y0 ] mmvd_ tile_group_ tile_group_ distance_fpel_mmvd_ fpel_mmvd_ idx enabled_ enabled_ [ x0 ][ y0 ] flag = = 0flag = = 1 0 1 4 1 2 8 2 4 16 3 8 32 4 16 64 5 32 128 6 64 256 7 128 512

Referring to Table 3 above, the distance of the MVD (i.e., MmvdDistance)is determined according to the value of the distance index (i.e.,mmvd_distance_idx), and the distance of the MVD (i.e., MmvdDistance) maybe derived by using integer sample precision or fractional sampleprecision based on the value of tilegroup_fpel_mmvd_enabled_flag. Forexample, when tile_group_fpel_mmvd_enabled_flag is equal to 1, itindicates that the distance of the MVD is derived by using integersample precision in the current tile group (or picture header), and whentile_group_fpel_mmvd_enabled_flag is equal to 0, it indicates that thedistance of the MVD is derived by using fractional sample precision inthe tile group (or picture header). In Table 1, information (flag) for atile group may be replaced with information for a picture header, forexample, tile_group_fpel_mmvd_enabled_flag may be replaced withph_fpel_mmvd_enabled_flag (or ph_mmvd_fullpel_only_flag).

In addition, the direction index (i.e., mmvd_direction_idx) indicatesthe direction of the MVD with respect to the starting point, and mayindicate four directions as shown in Table 4 below. In this case, thedirection of the MVD may indicate the sign of the MVD. The relationshipbetween the direction index and the MVD sign may be expressed as thefollowing table.

TABLE 4 mmvd_direction_ MmvdSign MmvdSign idx[ x0 ][ y0 ][ x0 ][ y0 ][0] [ x0 ][ y0 ][1] 0 +1 0 1 −1 0 2 0 +1 3 0 −1

Referring to Table 4, the sign of the MVD (i.e., MmvdSign) is determinedaccording to the value of the direction index (i.e.,mmvd_direction_idx), and the sign of the MVD (i.e., MmvdSign) may bederived for the L0 reference picture and the L1 reference picture.

Based on the above-described distance index (i.e., mmvd_distance_idx)and direction index (i.e., mmvd_direction_idx), the offset of the MVDmay be calculated using the following equations.

MmvdOffset[x0][y0][0=(MmvdDistance[x0][y0]<<2)*MmvdSign[x0][y0][0]  [Eq.2]

MmvdOffset[x0][y0][1]=(MmvdDistance[x0][y0]<<2)*MmvdSign[x0][y0][1]  [Eq.3]

In Equations 2 and 3, the MMVD distance (MmvdDistance[x0][y0]) and MMVDsigns (MmvdSign[x0][y0][0], MmvdSign[x0][y0][1]) may be derived based onTable 3 and/or Table 4. In summary, in the MMVD mode, a merge candidateindicated by a merge candidate flag (i.e., mmvd_cand_flag) is selectedfrom among the merge candidates in the merge candidate list derivedbased on the neighboring blocks, and the selected merge candidate isused as a base candidate (i.e., MVP). In addition, motion information(ie, motion vector) of the current block may be derived by adding an MVDderived using a distance index (i.e., mmvd_distance_idx) and a directionindex (i.e., mmvd_direction_idx) based on the base candidate.

A predicted block for the current block may be derived based on motioninformation derived according to the prediction mode. The predictedblock may include prediction samples (prediction sample array) of thecurrent block. When the motion vector of the current block indicates afractional sample unit, an interpolation procedure may be performed,through which prediction samples of the current block may be derivedbased on reference samples in the fractional sample unit within areference picture. When bi-prediction is applied, prediction samplesderived through weighting or weighted averaging (according to phase) ofprediction samples derived based on L0 prediction (that is, predictionusing reference picture and MVL0 in reference picture list L0) andprediction samples derived based on L1 prediction (that is, predictionusing reference picture and MVL1 in reference picture list L1) may beused as prediction samples of the current block. When bi-prediction isapplied, if the reference picture used for L0 prediction and thereference picture used for L1 prediction are located in differenttemporal directions with respect to the current picture (i.e.,bi-prediction and bidirectional prediction), it may be called truebi-prediction.

As described above, reconstructed samples and reconstructed pictures maybe generated based on the derived prediction samples, and thenprocedures such as in-loop filtering may be performed.

As described above, according to this document, when bi-prediction isapplied to the current block, prediction samples can be derived based ona weighted average. Conventionally, the bi-prediction signal (ie, thebi-prediction samples) can be derived through a simple average of the L0prediction signal (L0 prediction samples) and the L1 prediction signal(L1 prediction samples). That is, the bi-prediction samples were derivedas an average of the L0 prediction samples based on the L0 referencepicture and MVL0 and the L1 prediction samples based on the L1 referencepicture and MVL1. However, according to this document, whenbi-prediction is applied, a bi-prediction signal (bi-prediction samples)can be derived through a weighted average of the L0 prediction signaland the L1 prediction signal as follows.

In the above-described MMVD related embodiments, a method that considersa long-term reference picture in the MVD derivation process of MMVD maybe proposed, thereby maintaining and increasing compression efficiencyin various applications. In addition, the method proposed in theembodiments of this document can be equally applied to SMVD, which is asymmetric MVD technology used in inter mode (MVP mode), in addition tothe MMVD technology used in MERGE.

FIG. 9 is a diagram for describing a method of deriving motion vectorsin inter prediction.

In an embodiment of this document, it uses an MV derivation methodconsidering a long-term reference picture in a motion vector scalingprocess of a temporal motion candidate (a temporal motion candidate, atemporal merge candidate, or a temporal mvp candidate). The temporalmotion candidate may correspond to mvCol (mvLXCol). The temporal motioncandidate may be referred to as a TMVP.

The following table describes the definition of a long-term referencepicture.

TABLE 5   The function LongTermRcfPic( aPic, aPb, refIdx, LX), with Xbeing 0 or 1, is defined as follows:  If the picture with index refIdxfrom reference picture list  LX of the slice containing prediction blockaPb  in the picture aPic was marked as “used for long term  reference”at the time when aPic was the current  picture, LongTermRefPic( aPic,aPb, refIdx, LX) is equal to 1.  Otherwise, LongTermRefPic( aPic, aPb,refIdx, LX)  is equal to 0.

Referring to Table 5 above, if LongTermRefPic(aPic, aPb, refIdx, LX) isequal to 1 (true), the corresponding reference picture may be marked asused for long-term reference. For example, a reference picture notmarked as used for long-term reference may be a reference picture markedas used for short-term reference. In another example, a referencepicture not marked as used for long-term reference and not marked asunused may be a reference picture marked as used for short-termreference. Hereinafter, a reference picture marked as used for long-termreference may be referred to as a long-term reference picture, and areference picture marked as used for short-term reference may bereferred to as a short-term reference picture.

The following table describes the derivation of TMVP (mvLXCol).

TABLE 6   When availableFlagLXCol is equal to TRUE, mvLXCol andavailableFlagLXCol are derived as follows:   If LongTermRefPic( currPic,currCb, refIdxLX, LX ) is not equal to LongTermRefPic( ColPic, colCb,  refIdxCol, listCol ), both components of mvLXCol are set equal to 0and availableFlagLXCol is set equal   to 0.   Otherwise, the variableavailableFlagLXCol is set equal to 1, refPicList[ listCol ][ refIdxCol ]is set to be   the picture with reference index refIdxCol in thereference picture list listCol of the slice containing coding   blockcolCb in the collocated picture specified by ColPic, and the followingapplies:     colPocDiff = DiffPicOrderCnt( ColPic, refPicList[ listCol][ refIdxCol ] ) (8-402)     currPocDiff = DiffPicOrderCnt( currPic,RefPicList[ X ][ refIdxLX ] ) (8-403)     The temporal motion buffercompression process for collocated motion vectors as specified in    clause 8.5.2.15 is invoked with mvCol as input, and the modifiedmvCol as output.     if RcfPicList[ X ][ refIdxLX ] is a long-termreference picture, or colPocDiff is equal to currPocDiff,     mvLXCol isderived as follows:     mvLXCol = mvCol (8-404)     Otherwise, mvLXColis derived as a scaled version of the motion vector mvCol as follows:    tx = ( 16384 + ( Abs( td ) >> 1 ) ) / td (8-405)     distScaleFactor= Clip3 ( −4096, 4095, ( tb * tx + 32 ) >> 6 ) (8-406)     mvLXCol =Clip3( −131072, 131071, (distScaleFactor * mvCol +              128 − (distScaleFactor * mvCol >= 0 ) ) >> 8 ) ) (8-407)     where td and tbarc derived as follows:     td = Clip3( −128, 127, colPocDiff ) (8-408)    tb = Clip3( −128, 127, currPocDiff ) (8-409)

Referring to FIG. 9 and Table 6, when the type of the reference picturepointed to by the current picture (i.e., indicating whether a long-termreference picture (LTRP) or a short-term reference picture (STRP)) isnot equal to the type of the collocated reference picture pointed to bythe collocated picture, the temporal motion vector mvLXCol is not used.That is, when all of them are the long-term reference pictures or theshort-term reference pictures, colMV is derived, otherwise, colMV is notderived. In addition, in the case that all of them are the long-termreference pictures and in the case where the POC difference between thecurrent picture and the reference picture of the current picture is thesame as the POC difference between the collocated picture and thereference picture of the collocated picture, the collocated motionvector can be used as it is without scaling. If it is a short-termreference picture and the POC difference is different, the motion vectorof the collocated block is used after scaled.

In an embodiment of this document, MMVD used in the MERGE/SKIP modesignals a base motion vector index (base MV index), a distance index,and a direction index for one coding block as information for derivingMVD information. In the case of unidirectional prediction, MVD isderived from motion information, and in the case of bidirectionalprediction, symmetric MVD information is generated using a mirroring andscaling method.

In the case of bidirectional prediction, MVD information for L0 or L1 isscaled to generate an MVD of L1 or L0. However, when a long-termreference picture is referred, it requires modification in the MVDderivation process.

FIG. 10 illustrates an MVD derivation process of MMVD according to anembodiment of the present document. The method shown in FIG. 10 may befor a block to which bi-directional prediction is applied.

Referring to FIG. 10 , when the distance to the L0 reference picture andthe distance to the L1 reference picture are the same, the MmvdOffsetderived can be used as the MVD as it is, and the POC differences (POCdifference between the L0 reference picture and the current picture andthe POC difference between the L1 reference picture and the currentpicture) are different, MVD can be derived by scaling or simplemirroring (ie, −1*MmvdOffset) according to the POC difference andwhether it is a long-term or short-term reference picture.

In one example, a method of deriving symmetric MVD using MMVD for ablock to which bi-prediction is applied is not suitable for a blockusing a long-term reference picture. It is difficult to expectperformance improvement. Accordingly, in the following drawings andembodiments, an example is introduced, in which MMVD is not applied whenreference picture types of L0 and L1 are different.

FIG. 11 illustrates an MVD derivation process of MMVD according toanother embodiment of the present disclosure. The method shown in FIG.11 may be related to a block to which bi-directional prediction isapplied.

Referring to FIG. 11 , the MVD derivation method may vary according towhether a reference picture referenced by a current picture (or acurrent slice or a current block) is a long-term reference picture(LTRP) or a short-term reference picture (STRP). In one example, when amethod according to the embodiment of FIG. 11 is applied, part ofstandard specifications compliant with the present embodiment may bedescribed as shown in the following table.

TABLE 7   8.5.2.7 Derivation process for merge motion vector differenceInputs to this process are:   a luma location ( xCb, yCb ) of thetop-left sample of the current luma coding block relative to thetop-left   luma sample of the current picture,   reference indicesrefIdxL0 and refIdxL1,   prediction list utilization flags predFlagL0and predFlagL1. Outputs of this process are the luma merge motion vectordifferences in 1/16 fractional-sample accuracy mMvdL0 and mMvdL1. Thevariable currPic specifies the current picture. The luma merge motionvector differences mMvdL0 and mMvdL1 are derived as follows:   If bothpredFlagL0 and predFlagL1 are equal to 1, the following applies:     currPocDiffL0 = DiffPicOrderCnt( currPic, RefPicList[ 0 ][ refIdxL0] )      currPocDiffL1 = DiffPicOrderCnt( currPic, RefPicList[ 1 ][refIdxL1 ] )    If currPocDiffL0 is equal to currPocDiffL1 and , thefollowing applies:        mMvdL0[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]       mMvdL0[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]        mMvdL1[ 0 ] =MmvdOffset[ xCb ][ yCb ][ 0 ]        mMvdL1[ 1 ] = MmvdOffset[ xCb ][yCb ][ 1 ]    Otherwise, if Abs( currPocDiffL0 ) is greater than orequal to Abs( currPocDiffL1 ), the following    applies:        td =Clip3( −128,127, currPocDiffL0 )        tb = Clip3( −128, 127,currPocDiffL1 )        tx = ( 16384 + (Abs( td ) >> 1 ) ) / td       distScaleFactor = Clip3( −4096, 4095, ( tb * tx + 32 ) >> 6 )       mMvdL0[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]        mMvdL0[ 1 ] =MmvdOffset[ xCb ][ yCb ][ 1 ]      If RefPicList[ 0 ][ refIdxL0 ] is nota long-term reference picture and RefPicList[ 1 ][ refIdxL1 ]      isnot a long-term reference picture, the following applies:        mMvdL1[0 ] = Clip3( −2¹⁵, 2¹⁵ − 1, (distScaleFactor * mMvdL0[ 0 ] +        128− ( distScaleFactor * mMvdL0[ 0 ] >= 0 ) ) >> 8 )        mMvdL1[ 1 ] =Clip3( −2¹⁵, 2¹⁵ − 1, (distScaleFactor * mMvdL0[ 1 ] +        128 − (distScaleFactor * mMvdL0[ 1 ] >= 0 ) ) >> 8 )      Otherwise, IfRefPicList[ 0 ][ refIdxL0 ] is a long-term reference picture and     RefPicList[ 1 ][ refIdxL1 ] is a long-term reference picture, thefollowing applies:        mMvdL1[ 0 ] = Sign( currPocDiffL0 ) = = Sign(currPocDiffL1 ) ?                   mMvdL0[ 0 ] : −mMvdL0[ 0 ]      mMvdL1[ 0 ] = Sign( currPocDiffL0 ) = = Sign( currPocDiffL1 ) ?        mMvdL0[ 0 ] : −mMvdL0[ 0 ]       Otherwise, the followingapplies:        mMvdL0[ 0 ] = 0        mMvdL0[ 1 ] = 0        mMvdL1[ 0] = 0        mMvdL1[ 1 ] = 0    Otherwise (Abs( currPocDiffL0 ) is lessthan Abs( currPocDiffL1)), the following applies:        td = Clip3(−128, 127, currPocDiffL1 )        tb = Chp3( −128, 127, currPocDiffL0 )       tx = ( 16384 + ( Abs( td ) >> 1) ) / td        distScaleFactor =Clip3( −4096, 4095, (tb * tx + 32 ) >> 6 )        mMvdL1[ 0 ] =MmvdOffset[ xCb ][ yCb ][ 0 ]        mMvdL1[ 1 ] = MmvdOffset[ xCb ][yCb ][ 1 ]       IfRefPicList[ 0 ][ refIdxL0 ] is not a long-termreference picture and RefPicList[ 1 ][ refIdxL1 ]       is not along-term reference picture, the following applies:        mMvdL0[ 0 ] =Clip3( −2¹⁵, 2¹⁵ −1, (distScaleFactor * mMvdL1[ 0 ] +         128 −(distScaleFactor * mMvdL1[ 0 ] >= 0) ) >> 8 )        mMvdL0[ 1 ] =Clip3( −2¹⁵, 2¹⁵ − 1, , (distScaleFactor * mMvdL1[ 1 ] +         128 −(distScaleFactor * mMvdL1[ 1 ] >= 0) ) >> 8 ) )       Otherwise, IfRefPicList[ 0 ][ refIdxL0 ] is a long-term reference picture and      RefPicList[ 1 ][ refIdxL1 ] is a long-term reference picture, thefollowing applies:        mMvdL0[ 0 ] = Sign( currPocDiffL0 ) = = Sign(currPocDiffL1 ) ?                   mMvdL1[ 0 ] : −mMvdL1[ 0 ]       mMvdL0[ 1 ] = Sign( currPocDiffL0 ) = = Sign( currPocDiffL1) ?                  mMvdL1[ 1 ] : −mMvdL1[ 1 ]       Otherwise, thefollowing applies:        mMvdL0[ 0 ] = 0        mMvdL0[ 1 ] = 0       mMvdL1[ 0 ] = 0        mMvdL1[ 1 ] = 0   Otherwise ( predFlagL0or predFlagL1 are equal to 1 ), the following applies for X being 0 and1:       mMvdLX[ 0 ] = ( predFlagLX = = 1 ) ? MmvdOffset[ xCb ][ yCb ][0 ] : 0       mMvdLX[ 1 ] = ( predFlagLX = = 1 ) ? MmvdOffset[ xCb ][yCb ][ 1 ] : 0

FIG. 12 illustrates an MVD derivation process of MMVD according toanother embodiment of the present document. The method shown in FIG. 13may be for a block to which bi-directional prediction is applied.

Referring to FIG. 12 , a method for deriving MVD may be differentaccording to whether a reference picture referenced by a current picture(or a current slice, a current block) is a long-term reference picture(LTRP) or a short-term reference picture (STRP). In one example, whenthe method of the embodiment according to FIG. 12 is applied, a part ofthe standard document according to the present embodiment may bedescribed as shown in the following table.

TABLE 8 8.5.2.7 Derivation process for merge motion vector differenceInputs to this process are: - a luma location ( xCb, yCb ) of thetop-left sample of the current luma coding block relative to the top-left luma sample of the current picture,- reference indices refIdxL0 and refIdxL1,- prediction list utilization flags predFlagL0 and predFlagL1.Outputs of this process are the luma merge motion vector differences in 1/16 fractional-sample accuracy mMvdL0 and mMvdL1.The variable currPic specifies the current picture.The luma merge motion vector differences mMvdL0 and mMvdL1 are derived as follows: - If both predFlagL0 and predFlagL1 are equal to 1, the following applies:   currPocDiffL0 = DiffPicOrderCnt( currPic, RefPicList[ 0 ][ refIdxL0 ] )   currPocDiffL1 = DiffPicOrderCnt( currPic, RefPicList[ 1 ][ refIdxL1 ] ) - If (RefPicList[ 0 ][ refIdxL0 ] is a long-term reference picture and RefPicList  [ 1 ][ refIdxL1 ] is a short-term reference picture) or (RefPicList[ 0 ]  [ refIdxL0 ] is a short-term reference picture and RefPicList[ 1 ][ refIdxL1 ]   is a long-term reference picture), the following applies:     mMvdL0[ 0 ] = 0      mMvdL0[ 1 ] = 0      mMvdL1[ 0 ] = 0     mMvdL1[ 1 ] = 0  - Otherwise, the following applies:  - If currPocDiffL0 is equal to currPocDiffL1, the following applies:     mMvdL0[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]     mMvdL0[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]     mMvdL1[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]     mMvdL1[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]  - Otherwise, if Abs( currPocDiffL0 ) is greater than or equal to     Abs( currPocDiffL1 ), the following applies:     td = Clip3( −128, 127, currPocDiffL0 )     tb = Clip3( −128, 127, currPocDiffL1 )     tx = ( 16384 + ( Abs( td ) >> 1 ) ) / td     distScaleFactor = Clip3( −4096, 4095, ( tb * tx + 32 ) >> 6 )     mMvdL0[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]     mMvdL0[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]    • If RefPicList[ 0 ][ refIdxL0 ] is not a long-term     reference picture, the following applies:       mMvdL1[ 0 ] = Clip3( −2¹⁵, 2¹⁵ − 1, (distScaleFactor * mMvdL0[ 0 ] +       128 − ( distScaleFactor * mMvdL0[ 0 ] >= 0 ) ) >> 8 )       mMvdL1[ 1 ] = Clip3( −2¹⁵, 2¹⁵ − 1, (distScaleFactor * mMvdL0[ 1 ] +       128 − ( distScaleFactor * mMvdL0[ 1 ] >= 0 ) ) >> 8 )      • Otherwise, the following applies:       mMvdL1[ 0 ] = Sign( currPocDiffL0 ) = = Sign( currPocDiffL1 ) ?        mMvdL0[ 0 ] : −mMvdL0[ 0 ]       mMvdL1[ 0 ] = Sign( currPocDiffL0 ) = = Sign( currPocDiffL1 ) ?        mMvdL0[ 0 ] : −mMvdL0[ 0 ]     - Otherwise (Abs( currPocDiffL0 ) is less than      Abs( currPocDiffL1 )), the following applies:       td = Clip3( −128, 127, currPocDiffL1 )       tb = Clip3( −128, 127, currPocDiffL0 )       tx = ( 16384 + ( Abs( td ) >> 1 ) ) / td       distScaleFactor = Clip3( −4096, 4095, ( tb * tx + 32 ) >> 6 )       mMvdL1[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]       mMvdL1[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]      • If RefPicList[ 0 ][ refIdxL0 ] is not a long-term reference       picture, the following applies:       mMvdL0[ 0 ] = Clip3( −2¹⁵, 2¹⁵ − 1, (distScaleFactor * mMvdL1[ 0 ] +       128 − (distScaleFactor * mMvdL1[ 0 ] >= 0) ) >> 8 )       mMvdL0[ 1 ] = Clip3( −2¹⁵, 2¹⁵ − 1, , (distScaleFactor * mMvdL1[ 1 ] +       128 − (distScaleFactor * mMvdL1[ 1 ] >= 0) ) >> 8 ) )      • Otherwise, the following applies:       mMvdL10[ 0 ] = Sign( currPocDiffL0 ) = = Sign( currPocDiffL1 ) ?        mMvdL1[ 0 ] : −mMvdL1[ 0 ]       mMvdL0[ 1 ] = Sign( currPocDiffL0 ) = = Sign( currPocDiffL1 ) ?        mMvdL1[ 1 ] : −mMvdL1[ 1 ]  - Otherwise ( predFlagL0 or predFlagL1 are equal to 1 ),   the following applies for X being 0 and 1:      mMvdLX[ 0 ] = ( predFlagLX = = 1 ) ? MmvdOffset[ xCb ][ yCb ][ 0 ] : 0      mMvdLX[ 1 ] = ( predFlagLX = = 1 ) ? MmvdOffset[ xCb ][ yCb ][ 1 ] : 0

In summary, the MVD derivation process of MMVD, that does not deriveMVDs when the reference picture types in each direction are different,have been described.

In one embodiment according to the present document, MVD may not bederived in all cases of referencing a long-term reference picture. Thatis, when at least one L0 and L1 reference picture is a long-termreference picture, MVD is set to 0, and MVD can be derived only when ashort-term reference picture is included. It is described in detail inthe following drawings and tables.

FIG. 13 illustrates an MVD derivation process of MMVD according to anembodiment of the present document. The method shown in FIG. 13 may befor a block to which bi-directional prediction is applied.

Referring to FIG. 13 , based on the highest priority condition (RefPicL0!=LTRP && RefPicL1 !=STRP), MVD for MMVD may be derived when the currentpicture (or current slice, current block) refers to only short-termreference pictures. In one example, when the method of the embodimentaccording to FIG. 13 is applied, a part of the standard documentaccording to the present embodiment may be described as shown in thefollowing table.

TABLE 9 8.5.2.7 Derivation process for merge motion vector differenceInputs to this process are:- a luma location ( xCb, yCb ) of the top-left sample of the current luma coding  block relative to the top-left luma sample of the current picture,- reference indices refIdxL0 and refIdxL1,- prediction list utilization flags predFlagL0 and predFlagL1.Outputs of this process are the luma merge motion vector differences in 1/16 fractional-sample accuracy mMvdL0 and mMvdL1.The variable currPic specifies the current picture.The luma merge motion vector differences mMvdL0 and mMvdL1 are derived as follows:- If both predFlagL0 and predFlagL1 are equal to 1, the following applies:   currPocDiffL0 = DiffPicOrderCnt( currPic, RefPicList[ 0 ][ refIdxL0 ] )   currPocDiffL1 = DiffPicOrderCnt( currPic, RefPicList[ 1 ][ refIdxL1 ] ) - If RefPicList[ 0 ][ refIdxL0 ] is not a short-term reference picture or  RefPicList[ 1 ][ refIdxL1 ] is not a short-term reference picture, the following applies:     mMvdL0[ 0 ] = 0      mMvdL0[ 1 ] = 0      mMvdL1[ 0 ] = 0     mMvdL1[ 1 ] = 0  - Otherwise, the following applies:  - If currPocDiffL0 is equal to currPocDiffL1, the following applies:     mMvdL0[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]     mMvdL0[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]     mMvdL1[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]     mMvdL1[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]  - Otherwise, if Abs( currPocDiffL0 ) is greater than or equal to     Abs( currPocDiffL1 ), the following applies:     td = Clip3( −128, 127, currPocDiffL0 )     tb = Clip3( −128, 127, currPocDiffL1 )     tx = ( 16384 + ( Abs( td ) >> 1 ) ) / td     distScaleFactor = Clip3( −4096, 4095, ( tb * tx + 32 ) >> 6 )     mMvdL0[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]     mMvdL0[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]     mMvdL1[ 0 ] = Clip3( −2¹⁵, 2¹⁵ − 1, (distScaleFactor * mMvdL0[ 0 ] +     128 − ( distScaleFactor * mMvdL0[ 0 ] >= 0 ) ) >> 8 )        mMvdL1[ 1 ] = Clip3( −2¹⁵, 2¹⁵ − 1, (distScaleFactor * mMvdL0[ 1 ] +        128 − ( distScaleFactor * mMvdL0[ 1 ] >= 0 ) ) >> 8 )      - Otherwise (Abs( currPocDiffL0 ) is less than       Abs( currPocDiffL1 )), the following applies:        td = Clip3( −128, 127, currPocDiffL1 )        tb = Clip3( −128, 127, currPocDiffL0 )        tx = ( 16384 + ( Abs( td ) >> 1 ) ) / td        distScaleFactor = Clip3( −4096, 4095, ( tb * tx + 32 ) >> 6 )        mMvdL1[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]        mMvdL1[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]        mMvdL0[ 0 ] = Clip3( −2¹⁵, 2¹⁵ − 1, (distScaleFactor * mMvdL1[ 0 ] +        128 − (distScaleFactor * mMvdL1[ 0 ] >= 0) ) >> 8 )        mMvdL0[ 1 ] = Clip3( −2¹⁵, 2¹⁵ − 1, , (distScaleFactor * mMvdL1[ 1 ] +        128 − (distScaleFactor * mMvdL1[ 1 ] >= 0) ) >> 8 ) )  - Otherwise ( predFlagL0 or predFlagL1 are equal to 1 ),   the following applies for X being 0 and 1:    - If RefPicList[ X ][ refIdxLX ] is not a short-term     reference picture, the following applies:         mMvdLX[ 0 ] = 0        mMvdLX[ 1 ] = 0     - Otherwise, the following applies:       mMvdLX[ 0 ] = ( predFlagLX = = 1 ) ? MmvdOffset[ xCb ][ yCb ][ 0 ] : 0       mMvdLX[ 1 ] = ( predFlagLX = = 1 ) ? MmvdOffset[ xCb ][ yCb ][ 1 ] : 0

In the embodiment according to the present document, when referencepicture types in each direction are different, MVD is derived when ashort-term reference picture is obtained, and MVD is derived to 0 when along-term reference picture is included. It is described in detail inthe following drawings and tables.

FIG. 14 shows an MVD derivation process of MMVD according to anembodiment of the present document. The method shown in FIG. 14 may befor a block to which bi-directional prediction is applied.

Referring to FIG. 14 , when the reference picture types in eachdirection are different, MmvdOffset is applied when referring to areference picture (short-term reference picture) that is close to thecurrent picture, and MVD has a value of 0 when referring to a referencepicture (long-term reference picture) that is far from the currentpicture. In this case, a picture close to the current picture can beregarded as having a short-term reference picture, but when the closepicture is a long-term reference picture, mmvdOffset may be applied to amotion vector of a list indicating the short-term reference picture.

TABLE 10   mMvdL0 = 0 mMvdL1 = MmvdOffset mMvdL0 = 0mMvdL1 = (−1) * MmvdOffset mMvdL0 = MmvdOffset mMvdL1 = 0mMvdL0 = (−1) * MmvdOffset mMvdL1 = 0

For example, the four paragraphs included in Table 10 may sequentiallyreplace the lowermost blocks (contents) of the flowchart included inFIG. 14 .

In one example, when the method of the embodiment according to FIG. 14is applied, a part of the standard document according to the presentembodiment may be described as the following table.

TABLE 11 8.5.2.7 Derivation process for merge motion vector differenceInputs to this process are:- a luma location ( xCb, yCb ) of the top-left sample of the current luma coding  block relative to the top-left luma sample of the current picture,- reference indices refIdxL0 and refIdxL1,- prediction list utilization flags predFlagL0 and predFlagL1.Outputs of this process are the luma merge motion vector differences in 1/16 fractional-sample accuracy mMvdL0 and mMvdL1.The variable currPic specifies the current picture.The luma merge motion vector differences mMvdL0 and mMvdL1 are derived as follows:- If both predFlagL0 and predFlagL1 are equal to 1, the following applies:    currPocDiffL0 = DiffPicOrderCnt( currPic, RefPicList[ 0 ][ refIdxL0 ] )    currPocDiffL1 = DiffPicOrderCnt( currPic, RefPicList[ 1 ][ refIdxL1 ] ) - If currPocDiffL0 is equal to currPocDiffL1 and , the following applies:      mMvdL0[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]      mMvdL0[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]      mMvdL1[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]      mMvdL1[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ] - Otherwise, if Abs( currPocDiffL0 ) is greater than or equal to   Abs( currPocDiffL1 ), the following applies:      td = Clip3( −128, 127, currPocDiffL0 )      tb = Clip3( −128, 127, currPocDiffL1 )      tx = ( 16384 + ( Abs( td ) >> 1 ) ) / td      distScaleFactor = Clip3( −4096, 4095, ( tb * tx + 32 ) >> 6 )      mMvdL0[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]      mMvdL0[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]   - If RefPicList[ 0 ][ refIdxL0 ] is not a long-term reference    picture and RefPicList[ 1 ][ refIdxL1 ]     is not a long-term reference picture, the following applies:      mMvdL1[ 0 ] = Clip3( −2¹⁵, 2¹⁵ − 1, (distScaleFactor * mMvdL0[ 0 ] +      128 − ( distScaleFactor * mMvdL0[ 0 ] >= 0 ) ) >> 8 )      mMvdL1[ 1 ] = Clip3( −2¹⁵, 2¹⁵ − 1, (distScaleFactor * mMvdL0[ 1 ] +      128 − ( distScaleFactor * mMvdL0[ 1 ] >= 0 ) ) >> 8 )   - Otherwise, If RefPicList[ 0 ][ refIdxL0 ] is a long-term reference picture and     RefPicList[ 1 ][ refIdxL1 ] is a long-term reference picture, the following applies:      mMvdL1[ 0 ] = Sign( currPocDiffL0 ) = = Sign( currPocDiffL1 ) ?           mMvdL0[ 0 ] : −mMvdL0[ 0 ]        mMvdL1[ 0 ] = Sign( currPocDiffL0 ) = = Sign( currPocDiffL1 ) ?          mMvdL0[ 0 ] : −mMvdL0[ 0 ]       - Otherwise, the following applies:          mMvdL0[ 0 ] = 0         mMvdL0[ 1 ] = 0         mMvdL1[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]         mMvdL1[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]     - Otherwise (Abs( currPocDiffL0 ) is less than      Abs( currPocDiffL1 )), the following applies:         td = Clip3( −128, 127, currPocDiffL1 )         tb = Clip3( −128, 127, currPocDiffL0 )         tx = ( 16384 + ( Abs( td ) >> 1 ) ) / td         distScaleFactor = Clip3( −4096, 4095, ( tb * tx + 32 ) >> 6 )         mMvdL1[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]         mMvdL1[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]       - If RefPicList[ 0 ][ refIdxL0 ] is not a long-term        reference picture and RefPicList[ 1 ][ refIdxL1 ]        is not a long-term reference picture, the following applies:         mMvdL0[ 0 ] = Clip3( −2¹⁵, 2¹⁵ − 1, (distScaleFactor * mMvdL1[ 0 ] +         128 − (distScaleFactor * mMvdL1[ 0 ] >= 0) ) >> 8 )         mMvdL0[ 1 ] = Clip3( −2¹⁵, 2¹⁵ − 1, (distScaleFactor * mMvdL1[ 1 ] +         128 − (distScaleFactor * mMvdL1[ 1 ] >= 0) ) >> 8 ) )       - Otherwise, If RefPicList[ 0 ][ refIdxL0 ] is a long-term reference picture and        RefPicList[ 1 ][ refIdxL1 ] is a long-term reference picture, the following applies:         mMvdL10[ 0 ] = Sign( currPocDiffL0 ) = = Sign( currPocDiffL1 ) ?            mMvdL1[ 0 ] : −mMvdL1[ 0 ]         mMvdL0[ 1 ] = Sign( currPocDiffL0 ) = = Sign( currPocDiffL1 ) ?            mMvdL1[ 1 ] : −mMvdL1[ 1 ]       - Otherwise, the following applies:         mMvdL0[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]         mMvdL0[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]         mMvdL1[ 0 ] = 0          mMvdL1[ 1 ] = 0   - Otherwise ( predFlagL0 or predFlagL1 are equal to 1 ),    the following applies for X being 0 and 1:        mMvdLX[ 0 ] = ( predFlagLX = = 1 ) ? MmvdOffset[ xCb ][ yCb ][ 0 ] : 0        mMvdLX[ 1 ] = ( predFlagLX = = 1 ) ? MmvdOffset[ xCb ][ yCb ][ 1 ] : 0

The following table shows a comparison table between the examplesincluded in this document.

TABLE 12 Embodiment A Embodiment B Embodiment C Embodiment D L0  L1  L0 L1  L0  L1  L0  L1  L0 L1 POC Offset Offset Offset Offset Offset OffsetOffset Offset Short Short Same Offset Offset Offset Offset Offset OffsetOffset Offset Diff  Offset Scaled Offset Scaled Offset Scaled OffsetScaled (LO >=  L1) Diff  Scaled Offset Scaled Offset Scaled OffsetScaled Offset (LO <  L1) Long Long Same Offset Offset Offset Offset 0 0Offset Offset Diff  Offset (−1)  Offset (−1)  0 0 Offset (−1)  (LO >= Offset Offset Offset L1) Diff  (−1)  Offset (−1)  Offset 0 0 (−1) Offset (LO <  Offset Offset Offset L1) Short Long Same N/A N/A N/A N/AN/A N/A N/A N/A Diff  N/A N/A N/A N/A N/A N/A N/A N/A (LO >=  L1) Diff (−1)  Offset 0 0 0 0 Offset 0 (LO <  Offset L1) Long Short Same N/A N/AN/A N/A N/A N/A N/A N/A Diff  Offset (−1)  0 0 0 0 0 Offset (LO >= Offset L1) Diff  N/A N/A N/A N/A N/A N/A N/A N/A (LO <  L1)

Referring to Table 12, a comparison between methods of applying anoffset in consideration of reference picture types for MVD derivation ofMMVD described in the embodiments according to FIG. 10 to FIG. 14 isshown. In Table 12, Embodiment A may relate to the existing MMVD,Embodiment B may show the embodiment according to FIG. 10 to FIG. 12 ,Embodiment C may show the embodiment according to FIG. 13 , andEmbodiment D may show the embodiment according to FIG. 14 .

That is, in the embodiment according to FIG. 10 , FIG. 11 , and FIG. 12, the method of deriving MVD only when the reference picture types ofboth directions are the same has been described, and in the embodimentaccording to FIG. 13 , the method of deriving MVD only when bothdirections are short-term reference pictures has been described. In thecase of the embodiment according to FIG. 13 , MVD may be set to 0 in thecase of a long-term reference picture for unidirectional prediction. Inaddition, in the embodiment according to FIG. 14 , a method of derivingMVD in only one direction when reference picture types in bothdirections are different has been described. Differences between theembodiments represent various features of the techniques described inthis document, and it can be understood by those of ordinary skill inthe art that the effects to be achieved by the embodiments according tothis document can be implemented based on the features.

In an embodiment according to this document, when the reference picturetype is a long-term reference picture, a separate process is performed.When a long-term reference picture is included, the POC difference(POCDiff)-based scaling or mirroring does not affect performanceimprovement, so the MmvdOffset value is assigned to the MVD in thedirection having the short-term reference picture, and the value 0 isassigned to the MVD in the direction having the long-term referencepicture. In one example, when this embodiment is applied, a part of astandard document conforming to the present embodiment may be describedas shown in the following table.

TABLE 13 8.5.2.7 Derivation process for merge motion vector differenceInputs to this process are:- a luma location ( xCb, yCb ) of the top-left sample of the current luma coding  block relative to the top-left luma sample of the current picture,- reference indices refIdxL0 and refIdxL1,- prediction list utilization flags predFlagL0 and predFlagL1.Outputs of this process are the luma merge motion vector differences in 1/16 fractional-sample accuracy mMvdL0 and mMvdL1.The variable currPic specifies the current picture.The luma merge motion vector differences mMvdL0 and mMvdL1 are derived as follows:- If both predFlagL0 and predFlagL1 are equal to 1, the following applies: - If RefPicList[ 0 ][ refIdxL0 ] is a long-term reference picture or   RefPicList[ 1 ][ refIdxL1 ] is a long-term reference picture, the following applies:  - If RefPicList[ 0 ][ refIdxL0 ] is a short-term reference picture,    mMvdL0[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]    mMvdL0[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]     mMvdL1[ 0 ] = 0    mMvdL1[ 1 ] = 0  - Otherwise, if RefPicList[ 1 ][ refIdxL1 ] is a short-term reference picture,    mMvdL0 0 ] = 0     mMvdL0 1 ] = 0    mMvdL1[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]    mMvdL1[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]   - Otherwise,    mMvdL0[ 0 ] = 0     mMvdL0[ 1 ] = 0     mMvdL1[ 0 ] = 0    mMvdL1[ 1 ] = 0  - Otherwise, the following applies:   currPocDiffL0 = DiffPicOrderCnt( currPic, RefPicList[ 0 ][ refIdxL0 ] )   currPocDiffL1 = DiffPicOrderCnt( currPic, RefPicList[ 1 ][ refIdxL1 ] )  - If currPocDiffL0 is equal to currPocDiffL1, the following applies:    mMvdL0[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]    mMvdL0[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]       mMvdL1[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]       mMvdL1[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]     - Otherwise, if Abs( currPocDiffL0 ) is greater than or equal to       Abs( currPocDiffL1 ), the following applies:       td = Clip3( −128, 127, currPocDiffL0 )       tb = Clip3( −128, 127, currPocDiffL1 )       tx = ( 16384 + ( Abs( td ) >> 1 ) ) / td       distScaleFactor = Clip3( −4096, 4095, ( tb * tx + 32 ) >> 6 )       mMvdL0[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]       mMvdL0[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]       mMvdL1[ 0 ] = Clip3( −2¹⁵, 2¹⁵ − 1, (distScaleFactor * mMvdL0[ 0 ] +       128 − ( distScaleFactor * mMvdL0[ 0 ] >= 0 ) ) >> 8 )       mMvdL1[ 1 ] = Clip3( −2¹⁵, 2¹⁵ − 1, (distScaleFactor * mMvdL0[ 1 ] +       128 − ( distScaleFactor * mMvdL0[ 1 ] >= 0 ) ) >> 8 )     - Otherwise (Abs( currPocDiffL0 ) is less than      Abs( currPocDiffL1 )), the following applies:       td = Clip3( −128, 127, currPocDiffL1 )       tb = Clip3( −128, 127, currPocDiffL0 )       tx = ( 16384 + ( Abs( td ) >> 1 ) ) / td       distScaleFactor = Clip3( −4096, 4095, ( tb * tx + 32 ) >> 6 )       mMvdL1[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]       mMvdL1[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]       mMvdL0[ 0 ] = Clip3( −2¹⁵, 2¹⁵ − 1, (distScaleFactor * mMvdL1[ 0 ] +       128 − (distScaleFactor * mMvdL1[ 0 ] >= 0) ) >> 8 )       mMvdL0[ 1 ] = Clip3( −2¹⁵, 2¹⁵ − 1, , (distScaleFactor * mMvdL1[ 1 ] +       128 − (distScaleFactor * mMvdL1[ 1 ] >= 0) ) >> 8 ) )  - Otherwise ( predFlagL0 or predFlagL1 are equal to 1 ),   the following applies for X being 0 and 1:    - If RefPicList[ X ][ refIdxLX ] is a long-term reference picture      mMvdLX[ 0 ] = 0       mMvdLX[ 1 ] = 0     - Otherwise,      mMvdLX[ 0 ] = ( predFlagLX = = 1 ) ? MmvdOffset[ xCb ][ yCb ][ 0 ] : 0      mMvdLX[ 1 ] = ( predFlagLX = = 1 ) ? MmvdOffset[ xCb ][ yCb ][ 1 ] : 0

In another example, a portion of Table 13 may be replaced with thefollowing table. Referring to Table 14, Offset may be applied based on areference picture type other than POCDiff.

TABLE 14- If both predFlagL0 and predFlagL1 are equal to 1, the following applies: - If RefPicList[ 0 ][ refIdxL0 ] is a long-term reference picture or   RefPicList[ 1 ][ refIdxL1 ] is a long-term reference picture, the following applies:  - If RefPicList[ 0 ][ refIdxL0 ] is a short-term reference picture.   mMvdL0[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]   mMvdL0[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]   mMvdL1[ 0 ] = −MmvdOffset[ xCb ][ yCb ][ 0 ]   mMvdL1[ 1 ] = −MmvdOffset[ xCb ][ yCb ][ 1 ]  - Otherwise, if RefPicList[ 1 ][ refIdxL1 ] is a short-term reference picture,   mMvdL0 0 ] = −MmvdOffset[ xCb ][ yCb ][ 0 ]   mMvdL0 1 ] = −MmvdOffset[ xCb ][ yCb ][ 1 ]   mMvdL1[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]   mMvdL1[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]   - Otherwise,   mMvdL0[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]   mMvdL0[ 1 ] = MmvdOset[ xCb ][ yCb ][ 1 ]   mMvdL1[ 0 ] = −MmvdOffset[ xCb ][ yCb ][ 0 ]   mMvdL1[ 1 ] = −MmvdOffset[ xCb ][ yCb ][ 1 ]

In another example, a portion of Table 13 may be replaced with thefollowing table. Referring to Table 15, it is possible to always setMmvdOffset to L0 and -MmvdOffset to L1 without considering the referencepicture type.

TABLE 15- If both predFlagL0 and predFlagL1 are equal to 1, the following applies: - IfRefPicList[ 0 ][ refIdxL0 ] is a long-term reference picture or   RefPicList[ 1 ][ refIdxL1 ] is a long-term reference picture, the following applies:   mMvdL0[ 0 ] = MmvdOffset[ xCb ][ yCb ][ 0 ]   mMvdL0[ 1 ] = MmvdOffset[ xCb ][ yCb ][ 1 ]   mMvdL1[ 0 ] = −MmvdOffset[ xCb ][ yCb ][ 0 ]   mMvdL1[ 1 ] = −MmvdOffset[ xCb ][ yCb ][ 1 ]

According to an embodiment of this document, SMVD in the inter mode maybe performed similarly to MMVD used in the above-described MERGE mode.In the case of bidirectional prediction, whether or not symmetric MVD isderived is signaled from the encoding apparatus to the decodingapparatus, and when the related flag (ex. sym_mvd_flag) is true (or thevalue is 1), The second direction MVD (i.e., MVDL1) is derived throughmirroring of the first direction MVD (ex. MVDL0). In this case, scalingfor the first direction MVD may not be performed.

The following tables show syntaxes for decoding of a coding unit.

TABLE 16 coding_unit( x0, y0, cbWidth, cbHeight, treeType ) { Descriptor if( sps_smvd_enabled_flag && inter_pred_idc[ x0 ][ y0 ] = = PRED_BI &&  !inter_affine_flag[ x0 ][ y0 ] && RefIdxSymL0 > −1 && RefIdxSymL1 > −1 )  sym_mvd_flag[ x0 ][ y0 ] ae(v) if( inter_pred_idc[ x0 ][ y0 ] = PRED_L1 ) {  if( NumRefIdxActive[ 0 ] > 1 && !sym_mvd_flag[ x0 ][ y0 ] )   ref_idx_l0[ x0 ][ y0 ] ae(v)   mvd_coding( x0, y0, 0, 0 )  if( MotionModelIdc[ x0 ][ y0 ] > 0 )    mvd_coding( x0, y0, 0, 1 )  if(MotionModelIdc[ x0 ][ y0 ] > 1 )    mvd_coding( x0, y0, 0, 2 )  mvp_l0_flag[ x0 ][ y0 ] ae(v)  } else {   MvdL0[ x0 ][ y0 ][ 0 ] = 0  MvdL0[ x0 ][ y0 ][ 1 ] = 0  } if( inter_pred_idc[ x0 ][ y0 ] != PRED_L0 ) {  if( NumRefIdxActive[ 1 ] > 1 && !sym_mvd_flag[ x0 ][ y0 ] )   ref_idx_l1[ x0 ][ y0 ] ae(v)  if( mvd_l1_zero_flag && inter_pred_idc[ x0 ][ y0 ] = = PRED_BI ) {   MvdL1[ x0 ][ y0 ][ 0 ] = 0    MvdL1[ x0 ][ y0 ][ 1 ] = 0   MvdCpL1[ x0 ][ y0 ][ 0 ][ 0 ] = 0   MvdCpL1[ x0 ][ y0 ][ 0 ][ 1 ] = 0   MvdCpL1[ x0 ][ y0 ][ 1 ][ 0 ] = 0   MvdCpL1[ x0 ][ y0 ][ 1 ][ 1 ] = 0   MvdCpL1[ x0 ][ y0 ][ 2 ][ 0 ] = 0   MvdCpL1[ x0 ][ y0 ][ 2 ][ 1 ] = 0   } else {   if( sym_mvd_flag[ x0 ][ y0 ] ) {    MvdL1[ x0 ][ y0 ][ 0 ] = −MvdL0[ x0 ][ y0 ][ 0 ]    MvdL1[ x0 ][ y0 ][ 1 ] = −MvdL0[ x0 ][ y0 ][ 1 ]    } else    mvd_coding( x0, y0, 1, 0 )    if( MotionModelIdc[ x0 ][ y0 ] > 0 )    mvd_coding( x0, y0, 1, 1 )    if( MotionModelIdc[ x0 ][ y0 ] > 1 )    mvd_coding( x0, y0, 1, 2 )    mvp_l1_flag[ x0 ][ y0 ] ae(v)   } } else {   MvdL1[ x0 ][ y0 ][ 0 ] = 0   MvdL1[ x0 ][ y0 ][ 1 ] = 0  }... }

TABLE 17  if( sps_smvd_enabled_flag && inter_pred_idc[ x0 ][ y0 ] = = PRED_BI && !inter_affine_flag[ x0 ][ y0 ] && RefIdxSymL0 > −1 && RefIdxSymL1 > −1 ) sym_mvd_flag[ x0 ][ y0 ] ae(v)

Referring to Tables 16 and 17, when inter_pred_idc==PRED_BI andreference pictures of L0 and L1 are available (i.e., RefIdxSymL0>−1 &&RefIdxSymL1>−1), sym_mvd_flag is signaled.

The following table shows a decoding procedure for MMVD referenceindices according to an example.

TABLE 188.3.5 Decoding process for symmetric motion vector difference reference indicesOutput of this process are RefIdxSymL0 and RefIdxSymL1 specifying the list 0 and list 1 reference pictureindices for symmetric motion vector differences, i.e., when sym_mvd_flag is equal to 1 for a coding unit.The variable RefIdxSymLX with X being 0 and 1 is derived as follows:- The variable currPic specifies the current picture.- RefIdxSymL0 is set equal to −1.- For each index i with i = 0..NumRefIdxActive[ 0 ] − 1, the following applies: - When all of the following conditions are true, RefIdxSymL0 is set to i:  - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) > 0,  - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) <   DiffPicOrderCnt( currPic, RefPicList[ 0 ][ RefIdxSymL0 ] ) or RefIdxSymL0 is equal to −1.- RefIdxSymL1 is set equal to −1.- For each index i with i = 0..NumRefIdxActive[ 1 ] − 1, the following applies: - When all of the following conditions are true, RefIdxSymL1 is set to i:  - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) < 0,  - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) >   DiffPicOrderCnt( currPic, RefPicList[ 1 ][ RefIdxSymL1 ] ) or RefIdxSymL1 is equal to −1.- When RefIdxSymL0 is equal to −1 or RefIdxSymL1 is equal to −1, the following applies: - For each index i with i = 0..NumRefIdxActive[ 0 ] − 1, the following applies:  - When all of the following conditions are true, RefIdxSymL0 is set to i:   - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) < 0,   - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) >    DiffPicOrderCnt( currPic, RefPicList[ 0 ][ RefIdxSymL0 ] ) or RefIdxSymL0 is equal to −1. - For each index i with i = 0..NumRefIdxActive[ 1 ] − 1, the following applies:  - When all of the following conditions are true, RefIdxSymL1 is set to i:   - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) > 0,   - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) <    DiffPicOrderCnt( currPic, RefPicList[ 1 ][ RefIdxSymL1 ] ) or RefIdxSymL1 is equal to −1.

Referring to Table 18, a procedure for deriving availability ofreference pictures of L0 and L1 is described. That is, if there is areference picture in the forward direction among the L0 referencepictures, the index of the reference picture closest to the currentpicture is set as RefIdxSymL0, and the corresponding value is set as thereference index of L0. In addition, when if there is a reference picturein the backward direction among the L1 reference pictures, the index ofthe reference picture closest to the current picture is set asRefIdxSymL1, and the corresponding value is set as the reference indexof L1.

Table 19 below shows a decoding procedure for MMVD reference indicesaccording to another example.

TABLE 198.3.5 Decoding process for symmetric motion vector difference reference indicesOutput of this process are RefIdxSymL0 and RefIdxSymL1 specifying the list 0 and list 1 reference picture indices for symmetric motion vector differences, i.e., when sym_mvd_flag is equal to 1 for a coding unit.The variable RefIdxSymLX with X being 0 and 1 is derived as follows:- The variable currPic specifies the current picture.- RefIdxSymL0 is set equal to −1.- For each index i with i = 0..NumRefIdxActive[ 0 ] − 1, the following applies: - When all of the following conditions are true, RefIdxSymL0 is set to i:  - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) > 0,  - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) <   DiffPicOrderCnt( currPic, RefPicList[ 0 ][ RefIdxSymL0 ] ) or RefIdxSymL0 is equal to −1.- RefIdxSymL1 is set equal to −1.- For each index i with i = 0..NumRefIdxActive[ 1 ] − 1, the following applies: - When all of the following conditions are true, RefIdxSymL1 is set to i:  - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) < 0,  - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) >   DiffPicOrderCnt( currPic, RefPicList[ 1 ][ RefIdxSymL1 ] ) or RefIdxSymL1 is equal to −1.- When RefIdxSymL0 is equal to −1 or RefIdxSymL1 is equal to −1, the following applies: - For each index i with i = 0..NumRefIdxActive[ 0 ] − 1, the following applies:  - When all of the following conditions are true, RefIdxSymL0 is set to i:   - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) < 0,   - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) >    DiffPicOrderCnt( currPic, RefPicList[ 0 ][ RefIdxSymL0 ] ) or RefIdxSymL0 is equal to −1. - For each index i with i = 0..NumRefIdxActive[ 1 ] − 1, the following applies:  - When all of the following conditions are true, RefIdxSymL1 is set to i:   - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) > 0,   - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) <    DiffPicOrderCnt( currPic, RefPicList[ 1 ][ RefIdxSymL1 ] ) or RefIdxSymL1 is equal to −1.- When RefIdxSymL0 is not equal to −1 and RefIdxSymL1 is not equal to −1, the following applies - If (RefPicList[ 0 ][ RefIdxSymL0 ] is a long-term reference picture and RefPicList[ 1 ]  [ RefIdxSymL1 ] is a short-term reference picture) or   (RefPicList[ 0 ][ RefIdxSymL0 ] is a short-  term reference picture and RefPicList[ 1 ][ RefIdxSymL1 ] is a long-term reference picture),  RefIdxSymL0 and RefIdxSymL1 are set to −1.

Referring to Table 19, as in the embodiment described with FIG. 10 ,FIG. 11 , and FIG. 12 , when the types of L0 or L1 reference picturesare different, that is, if the reference picture types of L0 and L1 aredifferent after reference index derivation for SMVD, SMVD is not used inorder to prevent SMVD in a case that the long term reference picture andthe short term reference picture are used (refer to the lowermostparagraph of Table 19).

In an embodiment of this document, SMVD may be applied in the inter modesimilar to MMVD used in the merge mode. When a long-term referencepicture is used as in the embodiment described with FIG. 13 , thelong-term reference picture may be excluded from the reference indexderivation process for SMVD as shown in the following table in order toprevent SMVD.

TABLE 208.3.5 Decoding process for symmetric motion vector difference reference indicesOutput of this process are RefIdxSymL0 and RefIdxSymL1 specifying the list 0 and list 1 reference picture indices for symmetric motion vector differences, i.e., when sym_mvd_flag is equal to 1 for a coding unit.The variable RefIdxSymLX with X being 0 and 1 is derived as follows:- The variable currPic specifies the current picture.- RefIdxSymL0 is set equal to −1.- For each index i with i = 0..NumRefIdxActive[ 0 ] − 1, the following applies: - When all of the following conditions are true, RefIdxSymL0 is set to i:  - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) > 0,  - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) <   DiffPicOrderCnt( currPic, RefPicList[ 0 ][ RefIdxSymL0 ] ) or RefIdxSymL0 is equal to −1,  - RefPicList[ 0 ][ i ] is a short-term-reference picture.- RefIdxSymL1 is set equal to −1.- For each index i with i = 0..NumRefIdxActive[ 1 ] − 1, the following applies: - When all of the following conditions are true, RefIdxSymL1 is set to i:  - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) < 0,  - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) >   DiffPicOrderCnt( currPic, RefPicList[ 1 ][ RefIdxSymL1 ] ) or RefIdxSymL1 is equal to −1,  - RefPicList[ 0 ][ i ] is a short-term-reference picture.- When RefIdxSymL0 is equal to −1 or RefIdxSymL1 is equal to −1, the following applies: - For each index i with i = 0..NumRefIdxActive[ 0 ] − 1, the following applies:  - When all of the following conditions are true, RefIdxSymL0 is set to i:   - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) < 0,   - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) >    DiffPicOrderCnt( currPic, RefPicList[ 0 ][ RefIdxSymL0 ] ) or RefIdxSymL0 is equal to −1,   - RefPicList[ 1 ][ i ] is a short-term-reference picture. - For each index i with i = 0..NumRefIdxActive[ 1 ] − 1, the following applies:  - When all of the following conditions are true, RefIdxSymL1 is set to i:   - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) > 0,   - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) <    DiffPicOrderCnt( currPic, RefPicList[ 1 ][ RefIdxSymL1 ] ) or RefIdxSymL1 is equal to −1,   - RefPicList[ 1 ][ i ] is a short-term-reference picture.

The following table according to another example of this embodimentshows an example of processing not to apply SMVD when a long-termreference picture is used after reference picture index derivation forSMVD.

TABLE 218.3.5 Decoding process for symmetric motion vector difference reference indicesOutput of this process are RefIdxSymL0 and RefIdxSymL1 specifying the list 0 and list 1 reference pictureindices for symmetric motion vector differences, i.e., when sym_mvd_flag is equal to 1 for a coding unit.The variable RefIdxSymLX with X being 0 and 1 is derived as follows:- The variable currPic specifies the current picture.- RefIdxSymL0 is set equal to −1.- For each index i with i = 0..NumRefIdxActive[ 0 ] − 1, the following applies: - When all of the following conditions are true, RefIdxSymL0 is set to i:  - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) > 0,  - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) <   DiffPicOrderCnt( currPic, RefPicList[ 0 ][ RefIdxSymL0 ] ) or RefIdxSymL0 is equal to −1.- RefIdxSymL1 is set equal to −1.- For each index i with i = 0..NumRefIdxActive[ 1 ] − 1, the following applies: - When all of the following conditions are true, RefIdxSymL1 is set to i:  - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) < 0,  - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) >   DiffPicOrderCnt( currPic, RefPicList[ 1 ][ RefIdxSymL1 ] ) or RefIdxSymL1 is equal to −1.- When RefIdxSymL0 is equal to −1 or RefIdxSymL1 is equal to −1, the following applies: - For each index i with i = 0..NumRefIdxActive[ 0 ] − 1, the following applies:  - When all of the following conditions are true, RefIdxSymL0 is set to i:   - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) < 0,   - DiffPicOrderCnt( currPic, RefPicList[ 0 ][ i ] ) >    DiffPicOrderCnt( currPic, RefPicList[ 0 ][ RefIdxSymL0 ] ) or RefIdxSymL0 is equal to −1. - For each index i with i = 0..NumRefIdxActive[ 1 ] − 1, the following applies:  - When all of the following conditions are true, RefIdxSymL1 is set to i:   - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) > 0,   - DiffPicOrderCnt( currPic, RefPicList[ 1 ][ i ] ) <    DiffPicOrderCnt( currPic, RefPicList[ 1 ][ RefIdxSymL1 ] ) or RefIdxSymL1 is equal to −1.- When RefIdxSymL0 is not equal to −1 and RefIdxSymL1 is not equal to −1, the following applies - If RefPicList[ 0 ][ RefIdxSymL0 ] is a long-term reference picture or RefPicList[ 1 ][ RefIdx  SymL1 ] is a long-term reference picture, RefIdxSymL0 and RefIdxSymL1 are set to −1.

In one embodiment of this document, in the colMV derivation process ofTMVP, when the reference picture type of the current picture and thereference picture type of the collocated picture are different, themotion vector MV is set to 0, but the derivation method in the case ofMMVD and SMVD are not same as for TMVP and they need to be unified.

Even when the reference picture type of the current picture is along-term reference picture and the reference picture type of thecollocated picture is a long-term reference picture, the motion vectoruses the value of the collocated motion vector as it is, but MV may beset to 0 in MMVD and SMVD. In this case, TMVP also sets MV to 0 withoutadditional induction.

In addition, even if the reference picture types are different, along-term reference picture having a close distance to the currentpicture may exist. Therefore, instead of setting the MV to 0, colMV maybe used as the MV without scaling.

The following drawings are created to explain a specific example of thepresent specification. Since the names of specific devices described inthe drawings or the names of specific signals/messages/fields arepresented by way of example, the technical features of the presentspecification are not limited to the specific names used in thefollowing drawings.

FIGS. 15 and 16 illustrate a video/image encoding method and one exampleof a related component according to an embodiment(s) of the presentdisclosure. The encoding apparatus of FIG. 2 may perform the method ofFIG. 15 . Specifically, for example, the predictor 220 of the encodingapparatus may perform the S1500 to S1550 steps of FIG. 15 , and theresidual processor 230 of the encoding apparatus may perform the S1560step. The entropy encoder 240 of the encoding apparatus may perform theS1570 step. The method of FIG. 15 may include the embodiments of thepresent disclosure described above.

Referring to FIG. 15 , the encoding apparatus derives an interprediction mode for a current block S1500. Here, the inter predictionmode may include the merge mode, AMVP mode (the mode using motion vectorpredictor candidates), MMVD, and SMVD.

The encoding apparatus derives reference picture lists for the currentblock. In one example, the reference picture lists may include referencepicture list 0 (or L0, reference picture list L0) and reference picturelist 1 (or L1, reference picture list L1). For example, the encodingapparatus may configure reference picture lists for each slice includedin the current picture.

The encoding apparatus derives POC differences between each of thereference pictures included in the reference picture lists and thecurrent picture S1520. In one example, the POC difference between thecurrent picture and the previous reference picture of the currentpicture may be larger than 0. In another example, the POC differencebetween the current picture and the next reference picture of thecurrent picture may be smaller than 0. However, the description above isonly an example.

The encoding apparatus generates motion information including referencepicture indices for SMVD based on the POC differences S1530. Theencoding apparatus may derive reference indices for SMVD. The referenceindices for SMVD may indicate reference pictures for the SMVDapplication. The reference indices for SMVD may include reference indexL0 (RefIdxSumL0) and reference index L1 (RefIdxSumL1).

The encoding apparatus may construct a motion vector predictor candidatelist and derive a motion vector predictor based on the list. Theencoding apparatus may derive motion vectors based on symmetric MVDs andthe motion vector predictors. The motion information may include themotion vectors.

The encoding apparatus generates prediction samples based on the motioninformation S1540. The encoding apparatus may generate the predictionsamples based on motion vectors and the reference picture indicesincluded in the motion information. For example, the prediction samplesmay be generated based on the blocks (or samples) indicated by themotion vector among the blocks (or samples) within the referencepictures indicated by the reference picture indices.

The encoding apparatus generates prediction related information thatincludes the inter prediction mode S1550. The prediction relatedinformation may include information on MMVD and information on SMVD.

The encoding apparatus derives residual information based on theprediction samples S1560. Specifically, the encoding apparatus mayderive residual samples based on the prediction samples and the originalsamples. The encoding apparatus may derive residual information based onthe residual samples.

The encoding apparatus encodes the image/video information that includesthe prediction related information and residual information S1570. Theencoded image/video information may be output in the form of abitstream. The bitstream may be transmitted to the decoding devicethrough a network or a (digital) storage medium.

The image/video information may include various types of informationaccording to an embodiment of the present disclosure. For example, theimage/video information may include information disclosed in at leastone of Tables 1 to 21 described above.

In one embodiment, the reference picture indices for SMVD may be derivedbased on the short-term reference pictures included in the referencepicture lists.

In one embodiment, the image information may include information onmotion vector differences (MVD). The motion information may includemotion vectors. A first MVD for L0 prediction may be derived based onthe information on the MVD. A second MVD for L1 prediction may bederived based on the first MVD.

In one embodiment, the size of the second MVD may be the same as that ofthe first MVD. The sign of the second MVD may be opposite to that of thefirst MVD.

In one embodiment, the reference picture lists may include referencepicture list L0 for L0 prediction and reference picture list L1 for L1prediction. The POC differences may include a first POC differencebetween a first short-term reference picture included in the referencepicture list L0 and the current picture and a second POC differencebetween a second short-term reference picture included in the referencepicture list L1 and the current picture. The reference picture indicesmay include a first reference picture index indicating the firstshort-term reference picture and a second reference picture indexindicating the second short-term reference picture. The first referencepicture index may be derived based on the first POC difference. Thesecond reference picture index may be derived based on the second POCdifference.

In one embodiment, the first POC difference may be the same as thesecond POC difference.

In one embodiment, the reference picture lists may include the referencepicture list L0 for L0 prediction. The reference picture list L0 mayinclude a third short-term reference picture and a fourth short-termreference picture. The POC differences may include a third POCdifference between the third short-term reference picture and thecurrent picture and a fourth POC difference between the fourthshort-term reference picture and the current picture. Based on thecomparison between the third and fourth POC differences, the thirdshort-term reference picture may be derived as one of the referencepicture indices for SMVD.

In one embodiment, when the third POC difference is smaller than thefourth POC difference, the third short-term reference picture may bederived as one of the reference picture indices for SMVD.

FIGS. 17 and 18 illustrate an image/video encoding method and oneexample of a related component according to an embodiment(s) of thepresent disclosure. The decoding apparatus of FIG. 3 may perform themethod of FIG. 17 . Specifically, for example, the entropy decoder 310of the decoding apparatus may perform the S1700 step of FIG. 17 , andthe predictor 330 of the decoding apparatus may perform the S1710 toS1750 steps. The method of FIG. 17 may include the embodiments of thepresent disclosure described above.

Referring to FIG. 17 , the decoding apparatus receives/obtainsimage/video information S1700. The decoding apparatus may receive/obtainthe image/video information through a bitstream. The image/videoinformation may include prediction related information (includingprediction mode information) and residual information. The predictionrelated information may include information on MMVD and information onSMVD. Also, the image/video information may include various types ofinformation according to an embodiment of the present disclosure. Forexample, the image/video information may include the informationdisclosed in at least one of Tables 1 to 21 described above.

The decoding apparatus derives the inter prediction mode for a currentblock based on the prediction related information S1710. Here, the interprediction mode may include the merge mode, AMVP mode (the mode usingmotion vector predictor candidates), MMVD, and SMVD.

The decoding apparatus derives reference picture lists for the currentblock S1720. In one example, the reference picture lists may includereference picture list 0 (or L0, reference picture list L0) andreference picture list 1 (or L1, reference picture list L1). Forexample, the decoding apparatus may configure reference picture listsfor each slice included in the current picture.

The decoding apparatus derives POC differences between each of thereference pictures included in the reference picture lists and thecurrent picture S1730. In one example, the POC difference between thecurrent picture and the previous reference picture of the currentpicture may be larger than 0. In another example, the POC differencebetween the current picture and the next reference picture of thecurrent picture may be smaller than 0. However, the description above isonly an example.

The decoding apparatus derives motion information including referencepicture indices for SMVD based on the POC differences S1740. Thedecoding apparatus may derive reference indices for SMVD. The referenceindices for SMVD may indicate reference pictures for the SMVDapplication. The reference indices for SMVD may include reference indexL0 (RefldxSumL0) and reference index L1 (RefIdxSumL1).

The decoding apparatus may construct a motion vector predictor candidatelist and derive motion vector predictors based on the list. The decodingapparatus may derive motion vectors based on symmetric MVDs and themotion vector predictors. The motion information may include the motionvectors.

The decoding apparatus generates prediction samples based on the motioninformation S1750. The decoding apparatus may generate the predictionsamples based on motion vectors and the reference picture indicesincluded in the motion information. For example, the prediction samplesmay be generated based on the blocks (or samples) indicated by themotion vector among the blocks (or samples) within the referencepictures indicated by the reference picture indices.

In one embodiment, the reference picture indices for SMVD may be derivedbased on short-term reference pictures included in the reference picturelists.

In one embodiment, the image information may include information on MVD.The motion information may include motion vectors. A first MVD for L0prediction may be derived based on the information on the MVD. A secondMVD for L1 prediction may be derived based on the first MVD. The motionvectors may be derived based on the first and second MVDs.

In one embodiment, the size of the second MVD may be the same as that ofthe first MVD. The sign of the second MVD may be opposite to that of thefirst MVD.

In one embodiment, the reference picture lists may include referencepicture list L0 for L0 prediction and reference picture list L1 for L1prediction. The POC differences may include a first POC differencebetween a first short-term reference picture included in the referencepicture list L0 and the current picture and a second POC differencebetween a second short-term reference picture included in the referencepicture list L1 and the current picture. The reference picture indicesmay include a first reference picture index indicating the firstshort-term reference picture and a second reference picture indexindicating the second short-term reference picture. The first referencepicture index may be derived based on the first POC difference. Thesecond reference picture index may be derived based on the second POCdifference.

In one embodiment, the first POC difference may be the same as thesecond POC difference.

In one embodiment, the reference picture lists may include the referencepicture list L0 for L0 prediction. The reference picture list L0 mayinclude a third short-term reference picture and a fourth short-termreference picture. The POC differences may include a third POCdifference between the third short-term reference picture and thecurrent picture and a fourth POC difference between the fourthshort-term reference picture and the current picture. Based on thecomparison between the third and fourth POC differences, the thirdshort-term reference picture may be derived as one of the referencepicture indices for SMVD.

In one embodiment, when the third POC difference is smaller than thefourth POC difference, the third short-term reference picture may bederived as one of the reference picture indices for SMVD.

In the above-described embodiment, the methods are described based onthe flowchart having a series of steps or blocks. The present disclosureis not limited to the order of the above steps or blocks. Some steps orblocks may occur simultaneously or in a different order from other stepsor blocks as described above. Further, those skilled in the art willunderstand that the steps shown in the above flowchart are notexclusive, that further steps may be included, or that one or more stepsin the flowchart may be deleted without affecting the scope of thepresent disclosure.

The method according to the above-described embodiments of the presentdocument may be implemented in software form, and the encoding deviceand/or decoding device according to the present document is, forexample, may be included in the device that performs the imageprocessing of a TV, a computer, a smart phone, a set-top box, a displaydevice, etc.

When the embodiments in the present document are implemented insoftware, the above-described method may be implemented as a module(process, function, etc.) that performs the above-described function. Amodule may be stored in a memory and executed by a processor. The memorymay be internal or external to the processor, and may be coupled to theprocessor by various well-known means. The processor may include anapplication-specific integrated circuit (ASIC), other chipsets, logiccircuits, and/or data processing devices. Memory may include read-onlymemory (ROM), random access memory (RAM), flash memory, memory cards,storage media, and/or other storage devices. That is, the embodimentsdescribed in the present document may be implemented and performed on aprocessor, a microprocessor, a controller, or a chip. For example, thefunctional units shown in each figure may be implemented and performedon a computer, a processor, a microprocessor, a controller, or a chip.In this case, information on instructions or an algorithm forimplementation may be stored in a digital storage medium.

In addition, the decoding apparatus and the encoding apparatus to whichthe present disclosure is applied may be included in a multimediabroadcasting transmission/reception apparatus, a mobile communicationterminal, a home cinema video apparatus, a digital cinema videoapparatus, a surveillance camera, a video chatting apparatus, areal-time communication apparatus such as video communication, a mobilestreaming apparatus, a storage medium, a camcorder, a VoD serviceproviding apparatus, an Over the top (OTT) video apparatus, an Internetstreaming service providing apparatus, a three-dimensional (3D) videoapparatus, a teleconference video apparatus, a transportation userequipment (i.e., vehicle user equipment, an airplane user equipment, aship user equipment, etc.) and a medical video apparatus and may be usedto process video signals and data signals. For example, the Over the top(OTT) video apparatus may include a game console, a blue-ray player, aninternet access TV, a home theater system, a smart phone, a tablet PC, aDigital Video Recorder (DVR), and the like.

Furthermore, the processing method to which the present document isapplied may be produced in the form of a program that is to be executedby a computer and may be stored in a computer-readable recording medium.Multimedia data having a data structure according to the presentdisclosure may also be stored in computer-readable recording media. Thecomputer-readable recording media include all types of storage devicesin which data readable by a computer system is stored. Thecomputer-readable recording media may include a BD, a Universal SerialBus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, a magnetic tape, afloppy disk, and an optical data storage device, for example.Furthermore, the computer-readable recording media includes mediaimplemented in the form of carrier waves (i.e., transmission through theInternet). In addition, a bitstream generated by the encoding method maybe stored in a computer-readable recording medium or may be transmittedover wired/wireless communication networks.

In addition, the embodiments of the present document may be implementedwith a computer program product according to program codes, and theprogram codes may be performed in a computer by the embodiments of thepresent document. The program codes may be stored on a carrier which isreadable by a computer.

FIG. 19 shows an example of a content streaming system to whichembodiments disclosed in the present document may be applied.

Referring to FIG. 19 , the content streaming system to which theembodiment(s) of the present document is applied may largely include anencoding server, a streaming server, a web server, a media storage, auser device, and a multimedia input device.

The encoding server compresses content input from multimedia inputdevices such as a smartphone, a camera, a camcorder, etc. Into digitaldata to generate a bitstream and transmit the bitstream to the streamingserver. As another example, when the multimedia input devices such assmartphones, cameras, camcorders, etc. directly generate a bitstream,the encoding server may be omitted.

The bitstream may be generated by an encoding method or a bitstreamgenerating method to which the embodiment(s) of the present disclosureis applied, and the streaming server may temporarily store the bitstreamin the process of transmitting or receiving the bitstream.

The streaming server transmits the multimedia data to the user devicebased on a user's request through the web server, and the web serverserves as a medium for informing the user of a service. When the userrequests a desired service from the web server, the web server deliversit to a streaming server, and the streaming server transmits multimediadata to the user. In this case, the content streaming system may includea separate control server. In this case, the control server serves tocontrol a command/response between devices in the content streamingsystem.

The streaming server may receive content from a media storage and/or anencoding server. For example, when the content is received from theencoding server, the content may be received in real time. In this case,in order to provide a smooth streaming service, the streaming server maystore the bitstream for a predetermined time.

Examples of the user device may include a mobile phone, a smartphone, alaptop computer, a digital broadcasting terminal, a personal digitalassistant (PDA), a portable multimedia player (PMP), navigation, a slatePC, tablet PCs, ultrabooks, wearable devices (ex. Smartwatches, smartglasses, head mounted displays), digital TVs, desktops computer, digitalsignage, and the like. Each server in the content streaming system maybe operated as a distributed server, in which case data received fromeach server may be distributed.

Each server in the content streaming system may be operated as adistributed server, and in this case, data received from each server maybe distributed and processed.

The claims described herein may be combined in various ways. Forexample, the technical features of the method claims of the presentdocument may be combined and implemented as an apparatus, and thetechnical features of the apparatus claims of the present document maybe combined and implemented as a method. In addition, the technicalfeatures of the method claim of the present document and the technicalfeatures of the apparatus claim may be combined to be implemented as anapparatus, and the technical features of the method claim of the presentdocument and the technical features of the apparatus claim may becombined and implemented as a method.

1.-15. (canceled)
 16. An image decoding method performed by a decodingapparatus, the method comprising: obtaining image information includingprediction mode information and residual information from a bitstream;deriving a symmetric motion vector differences (SMVD) as a predictionmode for a current block based on the prediction mode information;deriving reference picture lists including L0 and L1 for the currentblock; deriving picture order count (POC) differences between each ofreference pictures and a current picture, for each of the L0 and the L1;deriving motion information including a reference picture index for eachof the L0 and the L1 based on the POC differences; generating predictedsamples of the current block based on the motion information; generatingresidual samples of the current block based on the residual information;and generating reconstructed samples of the current block based on thepredicted samples and the residual samples, wherein the referencepicture index for the L0 is derived based on a short-term referencepicture included in the L0, and wherein the reference picture index forthe L1 is derived based on a short-term reference picture included inthe L1.
 17. The method of claim 16, wherein the image informationincludes information on motion vector differences (MVD), wherein themotion information includes a motion vector for each of the L0 and L1,wherein a first MVD for the L0 is derived based on the information onMVD, and a second MVD for the L1 is derived based on the first MVD, andwherein the motion vector for the L0 is derived based on the first MVD,and the motion vector for the L1 is derived based on the second MVD. 18.The method of claim 17, wherein a size of the second MVD is the same asa size of the first MVD, and wherein a sign of the second MVD isopposite to a sign of the first MVD.
 19. The method of claim 16, whereinthe POC differences include a first POC difference for the L0 and asecond POC difference for the L1, wherein the first POC difference forthe L0 is a POC difference between a first short-term reference pictureincluded in the L0 and the current picture, and the second POCdifference for the L1 is a POC difference between a second short-termreference picture included in the L1 and the current picture, whereinthe reference picture index for the L0 indicates the first short-termreference picture and the reference picture index for the L1 indicatesthe second short-term reference picture, and wherein the referencepicture index for the L0 is derived based on the first POC difference,and the reference picture index for the L1 is derived based on thesecond POC difference.
 20. The method of claim 19, wherein the first POCdifference is the same as the second POC difference.
 21. The method ofclaim 16, wherein the L0 includes a third short-term reference pictureand a fourth short-term reference picture, wherein the POC differencesinclude a third POC difference between the third short-term referencepicture and the current picture and a fourth POC difference between thefourth short-term reference picture and the current picture, and whereinthe third short-term reference picture is derived as one of referencepicture indices for SMVD based on the comparison between the third POCdifference and fourth POC difference.
 22. The method of claim 21,wherein based on the third POC difference being smaller than the fourthPOC difference, the third short-term reference picture is derived as oneof the reference picture indices for SMVD.
 23. An image encoding methodperformed by an encoding apparatus, the method comprising: deriving asymmetric motion vector differences (SMVD) as a prediction mode for acurrent block; deriving reference picture lists including L0 and L1 forthe current block; deriving picture order count (POC) differencesbetween each of reference pictures and a current picture, for each ofthe L0 and the L1; deriving motion information including a referencepicture index for each of the L0 and the L1 based on the POCdifferences; deriving predicted samples of the current block based onthe motion information; generating prediction mode information relatedto the SMVD; deriving residual samples of the current block based on thepredicted samples; generating residual information related to theresidual samples; and encoding image information including theprediction mode information and the residual information, wherein thereference picture index for the L0 is derived based on a short-termreference picture included in the L0, and wherein the reference pictureindex for the L1 is derived based on a short-term reference pictureincluded in the L1.
 24. The method of claim 23, wherein the imageinformation includes information on motion vector differences (MVD),wherein the motion information includes a motion vector for each of theL0 and L1, wherein the information on MVD is related to a first MVD forthe L0, and wherein a second MVD for the L1 is derived based on thefirst MVD.
 25. The method of claim 23, wherein a size of the second MVDis the same as a size of the first MVD, and wherein a sign of the secondMVD is opposite to a sign of the first MVD.
 26. The method of claim 23,wherein the POC differences include a first POC difference for the L0and a second POC difference for the L1, wherein the first POC differencefor the L0 is a POC difference between a first short-term referencepicture included in the L0 and the current picture, and the second POCdifference for the L1 is a POC difference between a second short-termreference picture included in the L1 and the current picture, whereinthe reference picture index for the L0 indicates the first short-termreference picture and the reference picture index for the L1 indicatesthe second short-term reference picture, and wherein the referencepicture index for the L0 is derived based on the first POC difference,and the reference picture index for the L1 is derived based on thesecond POC difference.
 27. The method of claim 26, wherein the first POCdifference is the same as the second POC difference.
 28. The method ofclaim 23, wherein the L0 includes a third short-term reference pictureand a fourth short-term reference picture, wherein the POC differencesinclude a third POC difference between the third short-term referencepicture and the current picture and a fourth POC difference between thefourth short-term reference picture and the current picture, and whereinthe third short-term reference picture is derived as one of referencepicture indices for SMVD based on the comparison between the third POCdifference and fourth POC difference.
 29. The method of claim 28,wherein based on the third POC difference being smaller than the fourthPOC difference, the third short-term reference picture is derived as oneof the reference picture indices for SMVD.
 30. A non-transitory computerreadable storage medium storing a bitstream generated by a method,wherein the method comprising: deriving a symmetric motion vectordifferences (SMVD) as a prediction mode for a current block; derivingreference picture lists including L0 and L1 for the current block;deriving picture order count (POC) differences between each of referencepictures and a current picture, for each of the L0 and the L1; derivingmotion information including a reference picture index for each of theL0 and the L1 based on the POC differences; deriving predicted samplesof the current block based on the motion information; generatingprediction mode information related to the SMVD; deriving residualsamples of the current block based on the predicted samples; generatingresidual information related to the residual samples; and encoding imageinformation including the prediction mode information and the residualinformation to output the bitstream, wherein the reference picture indexfor the L0 is derived based on a short-term reference picture includedin the L0, and wherein the reference picture index for the L1 is derivedbased on a short-term reference picture included in the L1.